Python code to place trades with Liquid.com (a Japan) exchange – Part 2

In this article (continued from this one) I am going to play with Python a little bit. So, I will check whether a Python library installed previously (ccxt) is fit for purpose. I need to make some GET requests to Liquid.com’s API in order to retrieve some data and send some POST requests to place trades (sell orders or buy orders).

In the previous article (PART 1) I have achieved the followings:

  • I have successfully opened an account with the Liquid exchange where my trading experience started and created an API token (and a Secret Key) both to be used in my Python scripts;
  • Python was already installed on both a VPS (Virtual Private Server) and my laptop, and I have installed also the pip utility – useful to add Python open source code that generally is available on github;
  • and I have installed (using pip) the first open source library, i.e. ccxt – mentioned above i.e. a Python wrapper of REST API, these APIs being available for the majority of crypto exchanges (Liquid.com included).

Also, in this article I will try to define my trading strategy and see what other discounts this cryptocurrencies exchange (Liquid.com) has to offer.

Well, I will start with discounts: Liquid exchange has a huge discount at trading fees upon the condition to buy their crypto token named “Qash” (or coin? … whatever) and allow paying fees with it (I don’t know if Qash has its own blockchain to be a coin or it is a token issued on Ethereum chain, but it does not matter because traders can buy Qash using Liquid exchange). Below an excerpt taken from their site:

What role does Qash play?

All users are still able to receive a 50% discount on trading fees by paying with Qash. To pay your trading fees with Qash you must have a Qash balance in your Liquid account and turn on the option to pay fees with Qash in your account settings“.

So, one needs to buy a sufficient amount of Qash to cover the trading fees, configure the relevant setting as displayed above and bingo!

You got 50% discount on trading fees for all products 👍

Testing Liquid’s API using Python (ccxt library)

The code below lists the pairs existent on Liquid exchange:

import ccxt
exchange = ccxt.liquid()
exchange.loadMarkets()
listaPerechilor=list()
listaPerechilor=exchange.symbols

for i in listaPerechilor:
     print(i)

When I hit the run button, the above code produces the following output:

That is a list of symbols (I counted 383 elements but above you see an excerpt).

By “symbols” ccxt library means trading pairs.

Therefore, I have successfully tested the connection of my code to Liquid.com’s API and I can obtain (using the mentioned code snippet) all 383 pairs available at Liquid exchange.

If a trader wants to buy Bitcoin and uses his/her available US Dollars for that purpose (assuming this is done through calling the API like I am going to show in this series of articles), then the trader has to use the symbol (trading pair) “BTC/USD”.

To sell Bitcoin and receive the equivalent of US Dollars, the same pair is to be used (of course) but in the opposite direction.

In this series of articles when I write in code a pair to be used further to any of the following activities: (i) place orders, (ii) get an order status or (iii) cancel an order or (iv) update an order, etc etc (to enumerate at least a few basic operations in trading using API) it would be like below:

pair="BTC/USD"
# the variable pair will be used to communicate to Liquid API what 
# assets are to be bought or sold

Remark: the code snippet used to list the pairs did not use the API token because … it was not required. API token is something private to each and every trader and the list of pairs is public information.

On the contrary, placing orders using API is private. Therefore, such requests to Liquid’s API need to be authenticated first, meaning the API token created in the previous article (see this -> PART 1) will be used to (privately) manage your orders or checking your balances at Liquid.com exchange.

Available products at Liquid.com. Decisions, decisions … which one to pick?

Let’s analyze first two products:

  1. Spot Trading? Yes, but only when a crypto asset is going up. If you buy and then price goes lower the trader records a loss if he / she sells that asset. So, opportunities are only one way (price goes up) to make a profit.
  2. Margin Trading? As per this link (on Liquid website): “A margin trade is basically a bet, made with borrowed money, on 1) which way a product price will move and 2) how much it will move“.

I like more the “Margin Trading” product because I can take upon both opportunities, i.e. (i) when price goes up (ii) when price goes down. Is there any other possibility? 🤷‍♂️ Aah … yes: (iii) price doesn’t move.

Really? Nah … price will move. Especially in crypto. You now that joke? Daughter (blonde, sweet and persuading): Dad, I want to buy Bitcoin. Dad: Yes, sweety, of course. How much dollars do you need? Daughter: I don’t know, I just want to have 1 bitcoin as a souvenir (…). Dad: ok, one Bitcoin at …hold on … 35k dollars? So much? Why do you need to spend … (dad looking at the price chart) aaaa .. 36k dollars? I know someone who bought it for less, but now I have to pay … (chart doing its job) … aaaa … 38k $ ?!?

Plenty of volatility in crypto currencies … it’s bad for long term investors because … hmm .. well you don’t know if placing money today, until the year end an investor will gain or lose. Or until end of next year. It’s well known that bitcoin (and crypto market in general usually follows bitcoin) is so unpredictable … but for a speculative trader (this is one of my hobbies) it’s … quite a good chance to accummulate some profit! 👍

However, I am not greedy. I decided that I want to have 1% percent profit. Daily. I think in crypto assets market this is feasible. Won’t you think so?

If no, I understand. You say I am greedy.

Nah … I don’t give up.

If you don’t believe me this is achievable, please take a look at Bitcoin chart. It’s impossible not to take some opportunities for scalping that chart.

Problem is I don’t want to do a manual trading (I don’t have time), but what I am going to do is to start building a code aimed to be running 24/7 that is programmed beforehand to trigger a buy when a few pre-defined conditions are met.

If I buy (on Spot Trading) or I go long in Trading with Margin, after buying I expect a sell order will be placed also by that script. I expect my code to perform a sale when a profit is reached, or to sell at a pre-defined loss (let’s say at minus -15% referrenced to the buy level) if the target price is not reached and price goes so low.

And that’s all. Rinse and repeat.

Stop loss is a tool that is strongly recommended to be used if one is wise in trading, especially in crypto currencies. I want to protect my capital (although 2000 EUR is a small amount and this amount I can afford to lose, still I am going to do my best to not arrive with accounts emptied by bad trades).

Soooooo …… I studied some sources and looked at charts and … the pre-defined conditions to buy will be as follows:

  • RSI (Relative Strength Index): <= 25
  • Price increase: 0.1%

The above are two simple conditions that will trigger a buy order (when I will play in Spot and buy some crypto currency / asset or when I am going long on an asset in Margin Trading). They have to be both fulfilled (in the same time), or else there is no buy.

Rationale:

  • The basics of trading reveal that RSI indicator shows the underlying (crypto) asset (token or coin) is oversold when RSI is below 30 and is overbought when RSI is over 70. This is valid in any timeframe, no matter the trading pair.
  • The other condition, also valid for all pairs, i.e price to rise up (by 0.1% from the buy level – as compared to the previous candle) always does happen within a cycle. In fact, it happens all the time, but I am interested only when RSI enters the oversold range.

When RSI is going up (from oversold range) this means many traders start to buy, so price goes up quickly.

The second condition ensures buying at a lower price in the cycle. More precisely, I want to catch the moment at the change of trend, so when the price goes low, then RSI goes low too. Price continuing to go low, it comes a moment when price reaches the bottom of that cycle. At this moment (RSI being in oversold range) the price starts to rise and when it is going up by 0.1% (compared to the close of the previous candle) then … bam => buy order is placed.

If you look at majority of trading charts you might notice this pattern. When RSI approaches the bottom of oversold range (meaning price goes also at the bottom of that swing), if you wait a little bit more, there is a change. Then the price is going up quickly, so it is a very short window of opportunity to catch. A bot can help to get the job done and catch that moment (and buy earlier, i.e. when the price is yet no so high) by simply program it adequatelly.

So, the Python code will place a buy when the increase in price is yet a small increase, which I noticed watching charts that it is around 0.1%.

Selling part: hopefully, after placing the buy order the price will go up and the code I am going to build will place a sell order – let’s say – when price is +3% up (referenced to the buy level).

What is still to define – that I a have not yet decided – is the timeframe and the pair. But those are easy to code.

Notes:

  1. The above trading strategy should work either on both Spot Trading and Margin Trading. If one choses to trade with margin the difference between trading in Spot and trading with Margin is more profit / more loss when he/she trades with margin. This is because losses happen when market turns in the opposite direction of your trade and if the trader borrows money (this is the nature of Margin Trading product), the loss is larger. Vice-versa, i.e. more profit with Margin compared with Spot trading (when market “behaves”, i.e. the trade is going as planned) it’s also true because you borrow money and make profit also with those funds (besides the profit you make with your own funds).
  2. I know the trading strategy I am going to employ is a not a strong one. A professional trader expects a couple of conditions / indicators to be met plus another couple of other conditions / indicators to confirm a trend change before he/she becomes confident to buy, but I am good with it. So, I don’t expect my accout to go slim, at least not so soon (if it will ever do). But if so, then so be it. This experiment will stop. That’s it.
  3. Although this is not a super trading strategy, I expect to accummulate some profit in my account, but the most important target to be achieved is to have a 24/7 bot running and placing trades (some of them will be winning, some will be losing). Later on I will be tweakening this bot / code until it gives me a satisfactorily profit. If it will ever do that … ha ha ha …
  4. Improvement comes with experience in trading and I think I am gonna gain some experience during this exercise. Not much – because I will not allocate significant time for it (see this series of article that is published during week-ends), but some will mount (hopefully). I am prepared to take also some paid courses on internet – if I can make available more time for this. Finally, I hope at some point to have been added other conditions that need to be meet in order to trigger a buy/long or a sell/short. I am talking about – for example – 1) confirmations of trend change like MAs* crossover 2) other bullish flags like volume increase 3) bullish divergences 4) candlestick patterns and so on. For me the beauty of this experiment is more the automation of such trading, no matter the strategy employed.
  5. Let’s say I have success with the above strategy (hopefully) and first signs are good, profit is reached. I am aware that going further with any trading strategy and before to add more money into it, some tests will have to take place. So, at a later stage I am going to make my homework how this testing is done elsewhere and will employ some form of testing (or backtesting) in this project.

*) MA = Moving Averages. There are three MAs: 9 days, 21 days and 99 days used in trading stocks, but also in trading crypto currencies (I saw all of them defined as such in most crypto-currencies exchanges by default). Details will be provided at a later stage.

Coming back to the products existent in Liquid.com exchange. I look at that option no.2 (Margin Trading). This is with … borrowed money… Hmm … What is the interest on those borrowings? I want to know because an interest is a cost, so profit diminishes. But it shouldn’t be a fortune, right?

Let’s dig in to see exactly how high is this interest (will do this in PART 3).

So far I have completed the below steps:

  • I took advantage of reduced fees at Liquid exchange and I configured some setting in my account opened with Liquid in order to pay such fees with Qash (50% discount on trading fees …. yeeyee…) 👍
  • I started to assess the available products at Liquid exchange besides Spot Trading, namely Margin Trading (not decided though if I want to trade with margin or not) because fees are to be assessed (I will do it in the next article) and – most important – trading with high leverage is addicting and if you are wrong on a trade your account could be wiped out (liquidated) easily. Therefore, if I am going to trade with margin I’ll do it with minimum of leverage (I think 2x could be enough for this experiment – I will provide details as this series or articles unfolds) and with Stop Loss (of course).
  • I have build a small code snippet using ccxt library (a Python wrapper for APIs of 100+ crypto exchanges that includes Liquid.com’s API) that lists all the available pairs at this exchange
  • I have defined my trading strategy, not a perfect one, but subject of further improvements that hopefully will come along with gaining experience in trading, yet sufficient to get me motivated to achieve the final objective of this series of article: to automate trading by creating a bot that will trade 24/7 based on the underlying trading strategy. With good profits. Let’s hope for the best, but prepare for the worst (crypto is so volatile and a market in early stages of development that one needs to be very, very careful as many reported losses)
  • I did not place (not yet) any buy or sell order in this article because first I needed to design a trading strategy. Rushing is not my strong point, better to plan twice and execute once. A trading strategy is a must. I am comfortable with learning by doing and the readers might need to be patient enough … because they will be rewarded with a Python code that will be soon shown in its splendor … 🏆 … and will be accessible to all for free.

In PART THREE of this series of articles I am going to continue with this endeavor that is challenging and started already to motivate me because it looks … veeeeery promissing 👍

Python code to place trades with Liquid.com (a Japan) exchange

I wanted to start a trading experience using automation in Python, just for fun and to see if I suck at trading or I gain something. Normally, trading in cryptocurrencies is a high risk activity, but I have decided I can throw a couple thousands EURO in this experiment. Who knows, maybe I am good and I will increase my wealth from trading crypto. If not, at least I gain experience (?!?) 😁 using Python and a couple of things from that exchange.

Why Liquid.com?

In assessing various exchanges, I came across several not regulated (I don’t know why people trade there, they are shaddy – may be they think high risk – high return, so this might be fair enough for me, but I don’t like playing in a no man’s land anyway).

I have made this choice (Liquid) as I saw the following good points:

  • Official license from the Japan Financial Services Agency as a Virtual Currency Exchange.
  • Global Leading Fintech Innovators in 2019 no.38 (rated by KPMG)
  • Ranked consistently among the top 10 regulated cryptocurrency exchanges globally based on daily traded spot volume with deep BTC/JPY liquidity.
  • At its core Liquid is an institutional-grade exchange that caters to the needs of blockchain projects and consumers alike with access to new markets and fiat (fiduciary currencies, a.k.a. USD, JPY, etc) gateways.
  • Applied for the Major Payment Institution licence under the Singapore Payment Services Act for the provision of digital payment token services.

The experience in opening an account with them was normal (I got verified, they have Know Your Customer procedure in place – one cannot start trading without to get his/her account verified).

A serious exchange with a satisfactory customer service (English language I used to communicate with them without any problem). Government ID, an utility bill and setting up your bank account details (needed for deposit and withdrawals in dollars i.e. USD) have been the activities requested to start trading.

Python

I have installed Python (version 3.6.4) on both my laptop and VPS (a virtual private server). If one wonders why a VPS, there are a couple of arguments in its favor:

  1. availability of internet connection. When you buy a VPS service, the provider has a very good internet connection that is available 24/7 and it’s up 99,9% of time, including week-ends. Speed is needed to avoid delays in placing a trade. If market turns not in your favor, you might need to react quickly. Any interruption will cost you money. At home, you might interrupt the connection of scripts to internet when let’s say you forgot to plug in the laptop and battery is low (it happened to me quite often 🤷‍♂️)
  2. a server processes a Python Script more quickly than a laptop. So, I wanted to get data about prices in seconds and sometime on my laptop run many programms (Office with complex Excell sheets, Microsoft Access database, etc) and in addition if I watch a video from Youtube when learning some skills my laptop’s RAM gets overloaded and is running slow.

For installing Python one might want to check the steps I described in my previous post at this link (where you could find – if you are interested – also how to configure a VPS on Windows securely). I made that post when I wanted to ensure continuity of work in running some data collecting information from a couple of hundred websites using a script that take days (see also this link). Looking at that task when started, I noticed in Event Viewer of VPS some brute force attacks (unauthorised attempts to access my VPS from elsewhere), so I have secured that VPS and detailed at the mentioned link how I did it.

But it’s fine if you use the laptop just for a few number of tasks at the same time, the processing CPU is sufficient for trading if you don’t want to manage a database with complex algorhitms.

API (Application User Interface) at Liquid.com and how to use Python to access it

After one opens his account with Liquid exchange and is ready to trade, there is the possibility to create an API token.

API token comes with a secret key when created.

One can use these to trade at Liquid exchange not through its GUI (graphical user interface) i.e. a web browser, but using a script to connect to the REST API (for details regarding that API check this link).

Below there are some screen-shots for creation of an API token at Liquid.com exchange, as well as a screen shot from documentation (see the above-mentioned link).

Remarks:

  1. Strongly recommended is to keep that API token and Secret Key private. Anyone that has access to those two pieces of information can theoretically place trades and makes you bankrupt (unless you do it yourself … ha ha ha …)
  2. I have left unticked the withdrawal possibility using API. Unless you know what you are doing, strongly recommended is also to untick them.
  3. I left the IP restriction untouched. This is NOT recommended, but I have not decided yet where I want to trade from (tablet, VPS, laptop or phone). In case of tablet or phone usually the GSM provider cannot ensure the same IP when I use them. The IPs are changed upon criteria I cannot figure out. After I have a full automation (to code a script that will run from my laptop or 24/7 from my VPS), I will most likely add an IP and this means I cannot trade from other IPs except from that one that is added in Liquid settings.

At this link I have explained about the pip i.e. an utility that can aid at installing various Python libraries from github or from elsewhere (I like to integrate the work of others that aid in getting my job done more quickly – why to reinvent the wheel?).

So, I can simply install a new library from github with the following command at the Windows run -> cmd:

C:\WINDOWS\system32> pip install “opensource library”

and the relevant (open source) Python library gets installed on my laptop (Windows 10). Under the hood some files are automatically downloaded from github and installed in the relevant folders of my laptop. The process is transparent and quick.

For my purpose of trading at Liquid exchange I have installed ccxt (a library available for almost 100+ crypto exchanges), as follows:

C:\WINDOWS\system32>pip install ccxt

So far I have completed the below steps:

  • open an account with the relevant crypto exchange where my trading experience will start (Liquid.com)
  • (previously I have installed Python version 3.6.4 on both a VPS and my laptop, as well as the pip utility – useful to add Python open source code generally available on github)
  • and I have installed ccxt (a Python wrapper of REST API, these APIs being available for the majority of crypto exchanges that want to get volumes from traders that have some tools to automate trading, like me).

In PART TWO of this article I am going to start using these tools to place orders on Liquid exchange using a Python script.

Information Security Monitoring for companies rendering essential services is going to be closely watched by Romanian State

Deadline to submit the file to Romanian relevant authority is 17 December 2020.

It’s for companies in the industries offering essential services (mainly energy utilities, banking, payments services, etc)

Legal background:

Law 362/2018

Why deadline is 17 December 2020?

Because all related legal acts / governmental orders and so on have been culminated with HG 976/12.11.2020 published in Official Monitor as of 17 November 2020 and a stipulation says that legal documentation / complete file has to be submitted in 30 days (so, 30 days since 17 November 2020 is 17 December).

What if not complied with?

Companies have to make an analysis on their own and if qualify – according to the law – as providing / rendering essential services (mainly in industries as energy, banking, payment companies, etc), then they have to complete a file and submit it to relevant authority.

There are more to say, for example if the delivery of such services depend on a computer network and if they can suffer from a possible unavailability of such a network (in case of a security breach) then they qualify. If not dependant on a computer network to deliver the essential service yet they have to contact the relevant authority to be told if other criteria may make the company qualify for delivering essential services under the risk of unavailability in case of various events happen.

Pay attention: the penalties for not compliance are not small. So, if your company qualifies for the regulation, but if it is unaware about such legislation and does not do that internal analysis (and further to such analysis the conclusion is that it might be qualifying), the relevant State Authority may still check upon that company after deadline.

The fine is 5% of company’s turnover.

Despre Directoratul Național de Securitate Cibernetică – cîteva scăpări

Interesting opinion …

ADRIAN B. MUNTEANU

Scriu cele ce urmează după ce am lecturat de 3 ori propunerea de OG privind înființarea Directoratului Național de Securitate Cibernetică și pentru modificarea și completarea unor acte normative (la această oră nu știu dacă a fost aprobabtă). Am rezonat de prima dată de cînd am citit despre această inițiativă: este nevoie de un singur „creier„ care să deseneze viziunea, strategia, aspectele ce țin de conformitatea securității. Astăzi însă constat că, așa cum este scrisă propunerea, există posibilitatea să asistăm la nașterea unui monopol de stat („DNSC poate înființa, în condițiile legii, în numele statului, societăți cu capital de stat în România și în străinătate. De asemenea, DNSC poate participa, cu respectarea cadrului legal în vigoare din domeniul societar la majorarea capitalului social al societăților la care exercită, în numele statului, calitatea de acționar.) Sau poate înțelegerea mea este greșită în lipsa unei dezbateri publice pe acest proiect…

View original post 1,086 more words

A new digital assets exchange

Gets funding in a non-usual way

An IPO. Nothing new.

Business: provides services in trading digital assets. It was done already such a business.

Amount: 117 millions US dollars. It was done before, such amount is not unachievable.

So what is new?

SEC (Securities and Exchange Commission)-approved IPO is the first of its kind: investors can pay in Bitcoin (BTC), Ether (ETH), and USD Coin (USDC) – these are crypto-currencies.

Since launching the IPO, INX (the start-up company) raised $7.5 million by Sept. 10, the firm’s representatives said.

More than 3,000 retail and accredited investors registered for the INX token offering in the first three days.

Yes, that’s right. You read it correctly: a token. This is a new form of investment instead of shares or bonds or other equity form.

For those unfamiliar, a token is a form of digital assets that stays on a blockchain (a specific type of distributed databases network), is transferred back and forth over a blockchain and is issued electronically using a smart contract (a piece of software that is a feature of the blockchain).

So, no more need for papers, approvals, stamps, formalities and other similar stuff. Also, no more need for brokers. Investors can trade tokens using digital assests exchanges.

More precisely, this is a hybrid of IPO and security token offering (STO) that is registered with the SEC (a registration statement relating to the offering of these securities was declared effective by the SEC on Aug. 20), allowing everyday investors to legally participate in it.

The funds raised from the sale of INX tokens will be used to launch a multiservice digital asset platform. A regulated crypto trading platform for crypto, security tokens and their derivatives is intended to be created, as well as to launch a cash reserve fund.

STO (Securities Token Offering): Why is this a big deal anyway?

Because crowdfunding in a digital form takes place now in a regulated form.

Why was not regulated so far?

Because innovation was a step ahead of regulatory activities and this is since ever I would say. Blockchain as innovation makes no exception. Yes, the form not regulated was ICO (Initial Coin Offering)

In order to understand what an STO is, one must first understand ICO. The latter refers to a token offering from a company or organization in order to raise capital for a project. Buyers are issued with digital tokens. Unfortunately, ICOs are largely unregulated, thus putting investors at risk.

Sources:

https://cointelegraph.com/news/inx-exchange-launching-sec-registered-hybrid-token-share-ipo-this-week

https://101blockchains.com/sto-vs-ico-the-difference

Web scraping. Get products prices of competition from their website. How to achieve the goal using a trendy programming language (i.e. Python) in 20 lines of code (part 1)

Web scraping is when one wants to copy the content of a web page (or targeted parts of an entire site) and this is done automatically by a programmed robot (or a pre-programmed application). I use Python (and an open source library) to achieve successfully potentially any web scraping task.

An interesting use of web scraping to get automatically (hourly) the prices of products sold by competition

In this article I presume the reader has a business that sells to consumers (B2C) and, as any normal business, it needs to diligently monitor the competition, having available data about prices of its competitors. I presume also that each competitor has a website (eCommerce website, for example) that displays the product prices for each and every product they offer to consumers.

Disclaimer: it is debatable whether web scraping is legal or ilegal. A recent case showed that web scraping is not clearly legal, although legal issues such as copyright infrigements or contract law breaches were not addressed in the respective case of law (as per the author, i.e. Mr Eric Goldman, a professor at Santa Clara University School of Law, where he teaches and writes about Internet Law, Intellectual Property and Advertising Law). Therefore, before to pursue with web scraping, I strongly recommend the reader to consult his/her lawyer.

I decline any responsibility whatsoever for any endeavour the readers might pursue upon reading this article. Use web scraping at your own risk!

What I think it is still legal: no competitor can forbid you at watching. You open your browser, enter the URL of competitor’s website, you watch at figures and take a pencil and write down what prices that website displays for a specific product. But this is a manual approach that obviously is far from an efficient one.

Remark: the case of law displayed in the Disclaimer mentioned above dealt with an automated tool. Apparently, that tool was the root-cause of the problem because the target website stopped working.

Let’s clarify one thing: I would never recommend anything like that. Blocking websites from proper functioning is not in my intentions.

Now, that we passed the disclaimer part, I would like to say a few words about what this article provides. I explain a very convenient, cost effective and flexible method to get public info from the reader’s competitors’ websites (without disrupting any target website) and build a database to be analysed further. Such database aids any business to assess competition and improve internal decisions.

Thus, data gathered from competition or from market falls into “business intelligence” category. It is normal to make use of it when drafting the price strategy or set the size of discounts you are willing to offer to consumers (in order to make a difference compared to your main competitors), all aiming at maximising your sales (volume) and profit.

Advantage: gathering data from the market that goes into company’s database to be available to your analysts is what keep a business more adaptable to market. You might decide to reduce some prices to a category of products and/or increase prices for another products.

Manual approach won’t work for say hundred or thousand of products. If an important volume of market data from competitors is to be gathered, the process is less efficient, because you need additional staff to do it manually (or you can do it with less staff, but it is more time consuming).

A pre-programmed application (or a robot) aimed to extract the right data at the right time has the following advantages:

  • extract data with the frequency desired (it can be even hourly), so you are updated with prices practiced by competition all the time;
  • no manual error;
  • data obtained can be saved in your database or in the format needed (i.e. “csv” – for example) ready to be imported in your company’s database;
  • the bunch of data obtained would allow for various models to be created, various scenarios of prices modifications, with corresponding sales volumes and related profit by product and/or total profit maximisation.

Now, the technical part: what’s behind the scenes at the competition? Web pages and HTML

Any website has a bunch of web pages. When we visit a web page, our web browser makes a request to a web server that sends back some files that tell our browser how to render the page for us. 

The files our browser receives fall into a few main types:

  • HTML format / language – here we have the main content of the page.
  • CSS (Cascading Style Sheets) that add styling to make the page look nicer.
  • JS (Javascript) files that add interactivity to web pages.
  • Images — in some image formats, such as JPG, or PNG, etc that allow web pages to show pictures.

HTML is the main focus for web scraping (because it has the content that we target to obtain).

HyperTextMarkup Language (HTML) allows you to do similar things to what you do in a word processor like Microsoft Word, namely make text bold, create paragraphs, and so on. HTML is not as complex as Python.

HTML consists of elements called tags. Wherever you see “<” followed by one or several other words and then a similar “>” that ends those words, then these are called “tags“.

Examples:

<p> – indicates a beginning of a paragraph

“<a” – indicates a link then “>”. It is followed by the description of link (and finally followed by </a> at the end)

<body> – indicates the start of web page body

<head> – indicates the start of the header section of web page

and so on.

Why I entered into such details? Because usually, any web scraping tool uses these tags as the main elements needed to identify and extract the main content we are interested in.

Technical part of web scraping: as a tool used Python is the main actor

One of the most liked programming languages as per this link is Python.

In the next article I will continue with Python and a well known open-source module (or library) written in Python. I will explain how to create and approach web scrapping aimed at extracting the relevant content from your competitor’s website.

Customer Revenue Optimisation platform

Use-case: CenturyLink (this company is a member of the S&P 500 index and the Fortune 500)

An interesting use of AI (Artifical Intelligence) for increase revenue due to a better use of available data.

Due to a long history of mergers and acquisitions, CenturyLink (a telecom company now offering a bunch of digital services such as communications, network services, security, cloud solutions, voice, and managed services), created a number of information silos.

This most likely happens for companies that suffer such changes and the higher the number of such changes the higher the number of such information silos in the final company.

The drawback is that such silos prevent sellers from accessing the necessary account information about customers.

CenturyLink turned to Customer Revenue Optimisation (CRO) platform (name of provider is not important as I am not wrtiting this article to sell something specific).

“They (i.e. sales people) need AI insights to understand the kinds of triggers telling them which customer they should reach out to, and then look for other buying signals that can be reasons to go back and contact the customer. We like the prospect of leveraging activity as a way to give our frontline managers insight into what to do next, and AI helps drive that.”

Within three months of implementing the CRO platform, CenturyLink developed $250,000 a month in recurring revenue from one opportunity manager implementation, and a 350% funnel increase from top 40 accounts with account planning, according to https://www.computerweekly.com/news/252483148/CenturyLink-digitally-revamps-sales-framework

A major benefit of the implementation was the high level of accuracy it gave CenturyLink to make sales forecasts, which helped sellers to refocus on deals that were more attainable.

For more information about what IT solution you need that aid sales, is cost effective and suited to your specific situation, I would provide advise to your business (see my contact details)

SaaS businesses seem to have a good evolution at stock exchanges these times

A Software-as-a-Service company offers software from the cloud to its users

I touched the issue of SaaS (“Software as a Service”) in this post, when I discussed the delivery form of this software, looking more like a services and a little bit about differences compared to classic licensing model.

Companies that offer SaaS seem to be afloat the water in this new economic context.

As any novelty introduced by a new model in IT, this gets traction for various reasons, one of them might be because avoiding unnecessary costs since the value proposition of a SaaS could be maybe similar to a classical software application, but as a plus it includes also smaller fees for licensing than the classic model. In addition, nothing is delivered phisically (no CDs, no DVS, nor memory sticks with compiled installation executables – and therefore no headache with this bunch of stuff that is hard to control by the vendor not to be multiplied without a license by the buyer).

An understandable reason might be the job cuts that would indicate that many businesses might consider to reduce their costs, and since cloud computing looks like a cheaper alternative, this would be a way to minimise cashing out during this virus crisis. Although cloud costs – if not monitored – could add to the bill a significant amount also.

Source: https://techcrunch.com/2020/05/08/saas-stocks-defy-gravity-amid-pandemic-record-job-losses/

Python on a VPS (Virtual Private Server) for those who do not know Linux

A convenient way and secure enough to code in python language on Windows using a Virtual Private Server

VPS (Virtual Private Server) on Windows is nowadays a convenient use to code in Python if you don’t know Linux. Why a VPS? If you do not want to buy a computer, also if you want a speedy internet connection and no suden interruption of electrical power or forgot to save the work, with 20 $ a month you can have a VPS with a hosting provider.

Especially if you have some processing needs that might take longer than expected (I have some that takes more than 1 day), obviously to do it on laptop is not what you want. Even though a desktop placed at home that runs for more than 12 hours is exposed to suden interruptions when children go from a corner of the room to another, or the cleaning tasks might go over the wire … you know what I mean.

I have already a VPS (used on Windows Server 2012) since one year already and it works like a charm with python.

Note: please note that Windows Server 2008 will come to end of support in 2020, so you might want to choose some superior version of Windows Server .

Install and configure VPS

Access to the VPS is needed to be done using a VPN (Virtual Private Network) or other encrypted connection. In this short article I will go with VPN and I assume you have a hosting provider of VPS that offers VPN within its package.

Because on Windows, the RDP (Remote Desktop Protocol) is the standard to acces remotely a Windows computer, I have checked security of RDP and I am not in favor of it, without additional compensating security, i.e. VPN.

Only connect through VPN before to connect to VPS. THIS “before” IS A MUST. Otherwise, you will be exposed to hackers because Remote Desktop Protocol has flaws (see this link and this link in Romanian).

VPN is still good to go.

The next secure settings I have put in place for my VPS (after discussing also with one of my friends Adi Rusu – thank you Adi!). In my opinion, they are secure enough in my best reasonable judgement, based also on my experience of 12 years in IT field and having an engineering background, plus a CISA (Certified Information System Auditor) certification.

Please feel free to use them, but like anything in life do not take my word for free. Check for yourself and make your own judgement.

Therefore, use them at your own risk (DISCLAIMER: all these settings do not warrant you will be free from security incidents).

  • Create a user that is not admin (username not JIM, nor JOE, nor ADMINPOWER, etc, but something random like “Tdte59&eg0Y7)df6”)
  • Allocate only Remote Desktop rights to this new user. This is the only user that will be used for accessing the VPS
  • Session 24h for this user only (to keep the work going on 24/7 and the scripts to avoid be stopped by ending of session)
  • Create a second user with admin rights having random characters as a username (not ADMIN, nor ADMINISTRATOR, etc but something like “Rgdy6AGDTnr6el&5e” (or similar)). The longer the better.
  • Passwords at least 16 characters (special, upper case and numbers) to all users. The longer the better.
  • I saved these passwords (and also those long usernames) in a txt file on a memory stick that I attach to my laptop for small duration of time (like 30 seconds or so). I use that text file only when I go remote in order to access my VPS and want to connect. After I connect to VPS, the memory stick did its job and therefore I remove it. This makes my life easier (and hard to hackers). I will explain: convenient because I can use copy/paste from that memory stick when starting Remote Desktop Protocol to connect to VPS and do not need to remember such long and complex passwords. Hard for hackers: because (i) long passwords (ii) random usernames (hard to guess) avoiding usernames like ADMINISTRATOR for which they can do brute force attacks when do IP guessing and possibly find your IP (iii) my laptop is free of malware as it is configured as explained at last bullet, as to protect the credentials to access the VPS in two ways: no intruders allowed as no malware on the laptop and no credentials of VPS stored in my laptop anyway, but on a memory stick that is off line (in my pocket) (iv) and last, but not least: just in case a worst case scenario occur, i.e. say my laptop will get a malware (highly unlikely as I do not browse low quality sites and do not click sneaky attachments received to my email inbox), until the things go worst there are indicators that I am sensitive to, like slow processing, weird behaviour etc. So, I am confident will be able to notice in time and take appropriate actions (like stop using that laptop for accesing the virtual private server, disinfection, etc).
  • Pay attention: do not use that memory stick on public computers! Use it only on your personal computer after you are sure it is safe (no malware or virus on it – see last bullet below on how to achieve that)
  • Last step: the default ADMINISTRATOR account on Windows Server has to be deactivated (otherwise, if you do not configure properly the Windows Server, this account will be bombarded with brute force attacks from untraceable sources using Remote Desktop Protocol by hackers who might go random guessing your IP – I have searched the Event Viewer and saw those trials – so, deactivation of default admin account was the solution, as well as creation of another user account with admin rights having a random username – see fourth bullet point above)
  • Needless to say, your laptop (or home desk computer) that you use to access your VPS needs Anti-Virus and Firewall (I would strongly recommend to go for a paid solution and avoid free antivirus software like Avast – because, for example, data from Avast antivirus users was sold to advertisers, who can then combine it with other data they have on your activities to track you in great detail – see this link). Plus regular – at least weekly – full scans (of your laptop, not VPS). Last but not least, use a normal user (no admin rights) when you browse the world wide web from laptop. Even though there are temptations to use a full power user account on Windows (when I want to install something on my laptop and avoid installing errors), I log on using a normal user account I created. Thus, I have Windows 10 with a user that has no admin rights that I use all the time. Each time I need to install something on my laptop, that new application requires admin password, so no harm, I have control over browsing, as well as I can install whatever I want. These settings avoid a malware from a website I browse from laptop – that I do not know if it is malware or not – to be installed in the background, because if it is something that really needs to be malicious, it will ask for admin credentials. Therefore, if there is something that pops-up requring admin password and I don’t know what it is, I deny it.

That would be all regarding VPS (no need for anti-virus, firewall for server as long as you are invisible to the world wide web). No web server (IIS, Apache, etc) needed. On the contrary, I would advise to uninstall them if they are installed by default. So, this way your VPS will be invisible to hackers. But take care to NEVER browse world-wide web from the server (why would you do that? as long as you have the laptop already and can browse from it – and RDP allows for sharing the clipboard, you can transfer that information using copy from the browser of your laptop and paste onto the server – over RDP).

Install Python on Windows

Go on the server (Windows Server on your VPS).

I use version 3.6.4 (and Windows Server 2012 is on 64 bits) since more than one year.

The next link is to be used for downloading Python 3.6.4 (from official python.org) – always install from official sites:

Now, about “pip”:

I would highly recommend to install a tool that will always be needed for your convenience, when you want to benefit from open-source libraries written in python that are abundant on internet.

The next option is to be chosen when installing Python:

That would be all.

Now you have all you need to start scripting in python on Windows.

Libraries (like for example free work of others that usually is comprised in modules), depending on what you need will be downloaded and installed easily using pip (the optional feature above).

Usually, these librariries are installed from the command line (Powershell is also a possibility, instead of classic – inherited from DOS – command line).

Thus, to use classic command line, type “cmd” at start and right click “Run as Administrator”.

To use pip when installing python libraries available on github or elsewhere (where they specify pip as installing tool) type as follows from command line:

C:\WINDOWS\system32> pip installopensource library name that you want” (or whatever you want*)

You will notice a process that will follow in your command line, like a download of that library and additional components. The final outcome of that process should be finalised with success. This is the validation you have “downloaded” that library and you may start using it with python.

This “pip” and using command line are the most used features and also convenient for me when I explore various work in python made by others (usually they are free and could be found in a significant number on github).

Hope you will find enjoyable testing python on windows and convenient using a virtual private server (while you will be secure enough).

____________________________________________

*) Still be cautious about “whatever you want”. I mean about open-source code you find on the web (not everything is checked by open source communities as for us to have the comfort we are free from malicious bugs).

ChowNow platform partners with Instagram

Los Angeles restaurant ordering based platform will add “Order Food” buttons and stickers. Local restaurants aided this way

Food pictures and videos on Instagram are widely used. Users will be able to easily order because buttons and stickers will link directly to ChowNow to complete order flow.

It’s a great marketing tool for restaurants, said Chris Webb (ChowNow’s CEO and co-founder) and an easy way for them to inform their customers that they are open for business — even if they may not have open tables.

Nice idea, especially as restaurants were hardly hit by this virus crisis.

Source: https://techcrunch.com/2020/04/15/instagram-partners-with-las-chownow-to-make-food-pics-and-stories-shoppable/