Cruncher was updated so it will store only changes in publications’ views.
End users will not notice the difference, because missing days will be filled before rendering the chart.
The system will benefit with huge reduction in storage use and boosted performance when loading historical views data.
At the moment of writing this changelog, views data table occupied 73MB, and after the optimization it is 9.9MB, for 130000 publications!
A cosmetic change.
We used the word “Publish” initially because LBRY used it from the beginning. Now they are using the word “Upload”, but we dont like it, because one does not know if it is an item or an action of uploading the upload?! 🙂
The solution is to use word “Publication”, because on LBRY you can upload videos, images, audio, markdown files, and anything else, but you can also write articles which do not require you to upload anything.
Midnight script (that pulls view and follower numbers from LBRY API) got an additional code to handle errors, for example when, for whatever reason, LBRY API / ChainQuery does not respond correctly, the script will do the cleanup and start over after a brief cooldown period.
The cruncher that runs after this will be disabled and allowed to run again after the above script finishes properly.
In a continuous effort (struggle 😀 ) to improve performance using divide and conquer strategy, the script was updated to make bulk insert of publishes’ view data in bunches. It was bulk inserting before, but now it does it in bunches so it does not stress out server resources. More fine tuning might be needed to hit the sweet spot.
It will also ignore publishes without views, which will cut data storage in half! Yes, half of the 90k publishes from 408 channels being monitored by LBRYlytics users (at the moment of writing this news article) have ZERO views…
For you, the end user, this means you will not see data in your publishes’ views charts unless it actually starts getting some views 🙂
Midnight crunching script did not finish properly for 3 runs (August 04., 05. and 06.), which was really bad news.
Script was then improved (on August 6.) to execute multiple instances of itself as a background process, each instance processing at most 100 channels with all belonging publishes.
After monitoring it now during each run for last 4 days, it finished nicely all 4 times.
You can see historical performance reports on the info page.
Running this on a shared hosting is becoming programming art 🙂
You are now able to add multiple comma-separated wallet addresses to get more accurate reading of your holdings.
- custom homepage
- pin charts to homepage
First update to hopefully:
- improve loading times
- reduce database disk usage
Views are gathered every UTC midnight and stored in a database table, but for some weird reason *, I did not aggregate these from the begining to significantly reduce number of rows.
* bulk insert of 10k records is much faster than crunching, we want this script to run fast each midnight so we dont put a lot of stress on LBRY API, and process data later at slow pace 🙂
By significantly I mean reduce <N> days of history to only one row with all data gzdeflated, which will reduce select time and reduce storage space used. At the moment of writing this changelog, each 24 hours new 3MB of view data is stored. In less than a month the views table has grown to ~70MB. When crunching does its job and cleans up the raw view data table, that same data, but aggregated and gzipped, will occupy ~12MB, which is great improvement! It will grow much slower, for example instead of expected ~140MB it will be ~24MB etc.
This will also speed up displaying of your view stats, since we are not going to select N datapoints for N days from that huge table, but only one row for specific publish ID.
Crunching will sometimes happen when your publish view stats is displayed if there is uncrunched history, and in some other instances when page rendering does not require a lot of processing power and data queries.
Similar thing is coming for channel followers data, but it is not urgent since lbrylytics, at the moment of writing, processes 265 unique channels’ followers and 58102 publishes’ views. 21 days of history for each so far.
- daily channel followers charts
- daily publish views charts
- channel level tips
- improved accuracy for determining what is a tip and what is your own support
- you can add your local currency and it will appear on all charts
- channel grouping
- top publishes by earnings
- under the hood improvements
- settings page
- wallet address
- accordion settings
- added tip values for BTC and USD
- prices refreshed each minute from CryptoCompare API
- wallet address balance and value in BTC and USD
- dark styling
- chart improvements
- accordion-style channel list
- decision to go with Chart.js
- preparing data obtained from ChainQuery and rendering first tips per day chart!
- WordPress plugin base developed
- total tips obtained from LBRYio ChainQuery
- shortcode to show a channel with total tips