Paola Tomei, Author at Visokio

Improving Omniscope UI and UX

Paola Tomei — Thu, 17 Nov 2022 16:49:33 +0000

Steam is emerging from the Visokio labs – we’re busy adding a range of new features, enabling you to work better, smarter and achieve more with Omniscope… We’d love to share the news and let you know about the stuff you can test in the latest build…

App tour

Firstly – we wanted to revisit and improve the user experience – whether the user is an experienced analyst or a total novice. The app is more welcoming and offers help to discover the features, while at the same time facilitates a more intuitive interaction with different interfaces.

We’re rolling out the red carpet – our new users will appreciate an application tour popup – a dismissible sequence of hints to guide them through the basics on the Welcome screen, in the workflow app and on the report pages. Not a paperclip in sight!

Improved workflow building UX

The busy workflow builders will notice the new responsive magnetic auto-connection between the workflow blocks, when adding a new block or moving existing blocks around,

The whole blocks‘ palette can be kept open and pinned to the left, as a collapsible panel, to facilitate faster model building.

Add a new data source to an Omniscope workflow in two seconds: drag and drop a source file directly onto the workflow, or drop it onto a File block options screen to upload it to Omniscope.

Dealing with date patterns has gotten easier with the date format dropdowns in the Field Organiser and Report field settings, where the user can search for a pattern or enter a new one to the list.

Smooth creation of beautiful reports

Once all your charts are in place – why not give your dataviz a quick makeover?
When optimising the report layout for consumption on different device sizes, you can leverage multiple settings to create the best layout(s) for your report:

You can use the Free-form layout to position, resize, align and layer charts, filters and objects on the page.
The Multi page option is there when you need multiple scrollable pages in the same tab. The best choice for producing reports that look like documents.
Regarding the page sizing behaviour you can choose whether your dashboard will keep fixed size and ratio or to adapt and auto stretch to the screen size.

Free form layout

The perfect view proportions are achieved easily in the ‘Free form’ layout with the new snap views feature. (Select it from the Layout toolbar menu).

It is a user-friendly way to manipulate the charts on the page and create pixel-perfect dashboards by using chart sizing guide lines, while displaying both pixel count and the chart size as a % of the tab width.

Colours, fonts, borders, templates

The Report styles section now allows you to pick from a list of pre-configured style presets. You can also create your own presets and share them with colleagues as new Report styles (in XML format). The settings packed in these presets are data default colours, custom fonts, report background colours, elegant borders and spacing, as well as button and logo colours.

Projects page banner

Even the report list page got a bit of a makeover, with the ability to add banners with text, images, gifs and links using Markdown syntax.
Simply add a banner.md file to the projects folder. You will be able to add instructions or display a welcome message for the report consumers when they find themselves facing a long report list…
Read more

App translations

For our users in Asia we’ve improved “Simplified Chinese” translation, so they can interact with the translated menus and application components. We also support custom translations – read more here and let us know if you need help to implement. More info here.

Test and comment – let us know what you think!

The post Improving Omniscope UI and UX appeared first on Visokio.

Excel formula for success is +1: learn how to complement it with Omniscope

Paola Tomei — Thu, 19 May 2022 09:46:24 +0000

There are so many articles about the ‘Excel hell’, but the truth is people still love it. It’s easy to use and returns answers fast (at least for simple questions). It was not originally built for the multi-dimensional data gymnastics that form an ordinary day of today’s data analyst, so does that mean it’s time to drop Excel?

Not necessarily… Sometimes the winning formula will combine good old tools with innovative solutions. Omniscope provides an environment where any organisation can continue to use Excel for the tasks it’s good for and make it work as a productive component inside a modern data ecosystem. Xlsx and csv spreadsheets are useful data containers, which are easy to access and edit, but this convenience will inevitably take a toll on the company’s resources, if there is no automated ETL process to take the data from the source to a destination without the human intervention.

How to import, transform or blend Excel data in Omniscope?

a) Super speed: Importing Excel or csv spreadsheets in 2 seconds – just drag and drop a file onto the projects’ list page, or directly onto the data workflow canvas. Omniscope will load the file in the same folder and automatically visualise it! From there you can combine these inputs with data in other formats, from various sources (databases, live feeds or APIs…)

b) Live connection: connect to an Excel or csv spreadsheet on a local or network drive and preserve the connection. Browse to a file location from the File input block. You will be able to establish a live link, so any edits to the file in that location will be reflected and the new data will follow the same transformation process in Omniscope. This can happen on demand – by clicking on the ‘refresh button’ – or via an automated process, using the Scheduler application.

c) Octopus mode: connect to a folder and extract multiple files in one step! If the data storage mechanism collects multiple Excel/csv files in one folder there is no need to drag 52 weekly file blocks to the workflow to get the full picture of your annual data. The ‘Batch append folder’ block can facilitate the extraction of any number of Excel, csv or text files, adding the source field to show where each record came from. (Very useful for audit purposes!) By using the wild character the user can target specific file names or extensions.

d) Any file, any location – one block to append them all: Extract multiple files from different folders. The ‘Append Files’ block facilitates the extraction of multiple files from any location (machine, network or cloud location) and in any format (not just Excel – CSV, JSON, XML, etc). By default Omniscope will auto-detect the file type and read the contents of the file according to the default options for that file type. The user can configure how Omniscope will read the files by un-ticking ‘Auto detect file type’, then selecting the file type and further options.

Transformation phase – ETL and data quality evaluation

How can the analyst make sure that all those imported Excel files don’t contain any ‘fat finger’ entries, or a messy copy-paste result? Anyone who’s spent more than one Monday morning looking for a typo in a big table will understand that more important than getting the answers fast is making sure that those answers are correct.

You can combine multiple data quality tests inside an Omniscope workflow – start by adding a ‘Validate data’ block to your ETL process. It will screen the data schema (field list and data types), and also evaluate the record count, or the cell values in the target fields. If anything is wrong – a detailed data anomaly list will appear in the block’s diagnostics tab. You will not even have to worry how to remove duplicates – the operation is just a block away… And will be applied every time the new data is appended. More on data quality methods – read here.

Publishing the results

What happens after you’ve imported, validated, augmented and transformed spreadsheet data in Omniscope? You can customise and produce multiple outputs – share clean data with your team, as well as create interactive graphs to allow the users to explore the findings.

There are also options for mass publishing – for those serving hundreds of customised outputs. Use the ‘Batch Output’ block’s spreadsheet interface to define the filtering variation for the data outputs, or ‘Report Generator’ block to create dashboard variations e.g. showing just one country per report.

What about the price of this additional software? Excel is cheap… Think again – Excel is not automating your data management! The amount of man (and woman) hours lost on the manual data transformation / cleaning (copy/paste, rename, search/replace, pivot, then again?), and the time taken fishing for the data faults adds up to more cost than most modern data management tools. By far the biggest danger to an organisation is the cost of not having the right information at the right time or having incorrect data as a basis for decision-making!

Humble spreadsheets are here to stay, however, you may wish to make changes to how Excel is used and integrate it inside a modern data management system to allow it to excel!

The post Excel formula for success is +1: learn how to complement it with Omniscope appeared first on Visokio.

How to Visualise Geographic Time Data – Analysis of the Water Quality in Europe

Paola Tomei — Mon, 07 Jun 2021 16:01:39 +0000

The Findings

Are you only as good as your last performance? Analysis of the water quality in Europe in the year 2020 looks impressive – an assessment of coastline, lakes and transitional waters rated the majority of locations as “excellent”. Star country Cyprus achieved a 100% rating, while only two countries recorded less than impressive results.

Focusing on Poland’s performance we see that only 22% of the sampling locations are rated “excellent”, also that 401 data points out of 602 never submitted a measurement. With these records filtered out, the situation has changed: 66% of the submitted results are rated “excellent”, which is in line with the country’s results in the last 10 years. It is worth noting that Poland recorded dramatic improvement in 2011, jumping from just 24% to 67%. So – Poland’s result can be attributed to the data recording method, rather than an actual poor performance.

(Link to the interactive report)

On the other hand – the UK has shown notably poor records against the European neighbours in the last 10 years and the country has managed to record a performance over 80% only in 2013.

According to the Guardian “All English rivers have failed to meet quality tests for pollution amid concerns over the scale of sewage discharges and agricultural and industrial chemicals entering the water system.” Also that “water companies had poured raw sewage into rivers on more than 20,000 occasions in 2019, and dumped thousands of tonnes of raw sewage on beaches.”

Similarly to Poland, only a subset of the 640 registered measuring locations reported results in 2020 (183). When only those results are taken into account, the rating is 60%, which still leaves the UK at the bottom of the league.

This leads us to another question – how come that Great Britain, an island country with a coastline which is 12,429km long, has only 640 registered water quality measuring stations? Greece’s coast is 13,676km long an

d Italy’s coastline is about 7,500km long. Italians are measuring the quality of water in over 5.5K locations, France over 3.2K, Germany 2.2K and Greece in 1,634.

The most encouraging finding is that all countries (apart from the UK) have shown significant improvements over time, especially countries who joined the EU in the later wave, like Romania (where the proportion of ‘excellent’ ratings was increased from 6% to 70% in 13 years). EU membership candidate Albania has also improved from 51% in 2013 to 76% in 2020.

The UK has chosen to opt out of the EEA membership following Brexit, so this dataset will be the last one to include the UK and compare it to other European countries on a like-for-like basis.

How we made it: The data visualisation challenges

Every data visualisation has two phases: first the analyst needs to explore the data – to sort, compare, isolate, and identify relationships or trends. The second task is to communicate the findings effectively to an audience. Charts and tools used in the first ‘probing’ and exploratory phase are not necessarily the same visualisations to use for the story-telling presentation.

A challenge in this report was to create a comparison of results in the year 2020, on a geographical level, then provide perspective about the developments over time and explain how the countries got there. Is a good / bad result an outlier or is in line with a long-term trend?

From the macro level we can zoom into each individual performance and drill all the way down into the results for each measuring location. In two tabs we are moving from the continent/country aggregation down to your local beach/ river/ lake. We are going on a journey in space and time.

On the first tab the map with shapefiles aggregates the countries’ latest results and instantly identifies the two ‘offenders’ – a view that is complemented by the two bar charts: the first one ordering the % scores and another one adding context with regards to the number of measuring locations per country.

In a dataset with 30 countries and 30 years of data, it was useful to create a benchmarking effect and highlight one country of interest, while keeping the others in the frame grey – see the layered line view, which is driven by the Country Choice variable (configured in the Report Data Sources).

On the Country tab the viewer can follow the progress of one country at a time (filter choice limited to 1) and explore the spatial distribution of measuring places, as well as the performance over time on both the aggregated level and individual location level.

Data titles on both tabs are dynamic,

created by using Content view with integrated formulas, responsive to filtering; interactive heatmap is a Pivot chart with the cell values removed.

In this report not a single data point is wasted – curious viewers will be able to check the water quality before they dip their toes into the river, lake or the sea of their choice.

Londoners will have to satisfy themselves with the results for the Serpentine lake (rated ‘poor’ in 2020) and hope that Thames water will be rated soon!

Data transformation and preparation challenges

The dataset comes from the European Environment Agency: each record represents a measuring location, while each year of results was added as a new column. This is a straight-forward de-pivoting exercise, where we go from 43 fields x 22,276 records to 14 fields x 690K records! https://www.eea.europa.eu/themes/water/europes-seas-and-coasts/assessments/state-of-bathing-water/state-of-bathing-waters-in-2020

Another challenge was to handle the 3 location Management status fields (Management2018, Management2019, Management2020), which were applicable and populated only in the last 3 years. De-pivoting them with the rest of the dataset was not an option. These fields were therefore isolated, de-pivoted on their own, then merged with the main dataset, so they joined only the relevant records (merging on both location ID and Year fields).

Data transformation depends on the visualisation requirements – once the basics are done (field formatting, cleaning, classification, validation) requirements will come from the visualisation. Is the data orientation suitable for the charts, what is the data granularity required for the analysis? Are all the fields in the dataset relevant? The cool thing about Omniscope is that the ETL (transformation) and visualisation components go hand-in-hand. The visualisation can come first and be used for the data diagnostic purposes, to decide on the course of action, then again at the end, for the data presentation. The analyst can seamlessly go back and forth, flip between the two modes, and make changes to the underlying data, even while in the middle of the visualisation job, to quickly shorten the label, change a data field format or add a new calculation.

The post How to Visualise Geographic Time Data – Analysis of the Water Quality in Europe appeared first on Visokio.