FlowSphera 1.17.0 release notes

Long periods from the summary tables. A day or more of a busy network no longer reads every record. With 216 million records in the day on screen (12 hours at 5,000 records/s) all the pages' requests together answer 4× faster, the slow pages 4–19× (Overview 35 s → 3 s, Flows 31 s → 3 s), with the same numbers: each summary table was checked row by row against the records.

FlowTrack by TELESPHERA: the new logo — the TELESPHERA globe, turning slowly (still when the system asks for less motion), with «by TELESPHERA» in the sidebar and on the sign-in page; the browser tab shows the globe too.
A busy network's Flows page over days: Top conversations and the river of a whole period read the new conversations per hour (inside host × outside address) instead of every record — on 2,200 records/s the heaviest query of a day went from 25 s to under 1 s. The table fills its history by itself after the upgrade, newest day first.
Every page over a day or more reads the summary tables, not the records (the rest of the same work):
- QoS reads its own totals per DSCP class — they were built since 1.16.0 but never used.
- Ports: the inside hosts and the outside addresses of each port are counted exactly from new hourly tables, instead of from the records of the period.
- A filter by service, country, city or AS, a filter by an inside host, the inside traffic (hosts, networks, the chart) and the totals at the top of a page stay on the summary tables too; before, each of them fell back to the records.
- A period of a day or more starts and ends on a whole hour, so every summary table can answer it; a period that ends now takes its last minutes from the 5-minute totals.
- The bell's «now» events (a burst from one host, a first contact with a new country) read the summary tables and are worked out once a minute for every open tab together, not once per tab.
- A port filter (e.g. port:443) keeps Top conversations, the river and Path analysis on the summary tables too: a new hourly table of the conversations per port is read only under a port filter, so the pages without one stay as fast as they are. The AS of a port's traffic (Services) comes from the per-port totals as well.
- A summary table whose definition changes in an upgrade is filled again alone; the others keep their history.
A chosen host (ip: filter) over inside or all traffic reads the summary tables too. The sessions in which the host was the server are turned round to it from the conversations per hour, exactly as from the records — on 216 million records in a day (12 hours at 5,000 records/s): the host page 34 s → 6–8 s, the totals 15 s → 1 s, Top hosts 11 s → 0.8 s, its services 6.6 s → 0.5 s and protocols 4.2 s → 0.6 s, the river 20 s → 1.2 s, Through device paths 14 s → 0.6 s. Ports, QoS and Geolocation of a chosen host still read its records.
A chosen host over inside or all traffic: the comparison with the period before (the arrows on the totals) was always missing — the host's records were cut at the start of the period on screen.
Through device: the interface load of a big firewall in under a second (was ~20 s for a day): on a firewall with 250 interfaces (63,000 in/out pairs) the totals came back per PAIR of interfaces, 6.4 million rows a day for the web service to add up; each record is now split between its two interfaces in the database. The same figures.
Through device: a device without traffic in the period left Paths and Interfaces waiting for ever; they now say there is no traffic, as the map does.
Every page waited ~8 s on a firewall with many interfaces: Path analysis works out which devices are neighbours every 10 minutes in the background, and it looked each address of the last hour up in every network one by one — on a 300-interface firewall at 5,000 records/s that was 1½–3 minutes of CPU, during which the web service answered every page slowly (Path analysis and Through device ~8 s even for an hour). The lookup is indexed now: the same hour takes about a second, with exactly the same result.
«database error» on long periods: the web service gave up on a query after 30 s while ClickHouse was let run it for 2 minutes — a slow page failed with that error while the query went on loading the server. The service now waits as long as ClickHouse may run it, and a query whose page is gone stops.
Path analysis over days: tracing a source to a destination over a period reads the conversations per hour (which now also keep each device's interfaces and the address after NAT) instead of every record of the period; only its first and last partial hours and live windows read the records. The hops show the same bytes, records and conversations (inside host × outside address, as on Flows) — counted exactly.
Detections learn without reading the history again: every hour the detector re-learned the services of the last day and the DDoS baselines of the last week from every record (at 5,000 records/s ~3 billion records an hour). It now keeps what it needs as it looks at each 10-minute window — the minutes of traffic per device and per service, and the candidates of services per hour — in small tables, and learns from those; the first start after the upgrade fills them once from the records. The baselines are now exact (before: an approximate percentile that differed a little from run to run).
Sustained upload without 3 hours of records a minute: the bell and the Events page asked every minute which inside hosts sent over 5 Mbit/s to one address for 45 minutes of the last 3 hours, reading all those hours of records (at 5,000 records/s 54 million). The detector now keeps those busy minutes as it looks at each window; only the minutes since its last window are read from the records. The same events, found the same way.
The summary tables are filled ~5× faster and come out smaller: filling the history after an upgrade summed the records up per second instead of per 5 minutes / per hour — every number was right, but the tables held many more rows than needed (the 5-minute ones up to 300×) until ClickHouse merged them. Now each bucket is one row, and on a server with 6 or more cores two or three tables are filled at a time (14 tables, an hour of 8 M records on 6 cores: 100 → 21 s), each in the same memory as before. What is received after the upgrade was always summed up right.