22 Apr Omniscope Rock Build – 2022.1

They say good things come to those who wait… Because our new Omniscope 2022.1 Rock release took slightly longer this time – it is only right that we treat you to more goodies than usual… Whatever it is that you wish to do this spring: to clean, build, move and improve – you’ll find that Omniscope has just the right tools to do it.

Bring in the Robots!

Save time with the Macro functionality – create reusable workflow logic, then add it as an ETL routine to other projects. 

The new Rock build adds the ability to decompose complex workflows into smaller “macros” and optionally share them with other users/ folders/ projects on the server via the Macros folder.

You can use macros without sharing them, as a way of reducing the complexity of a large workflow, by creating a smaller project alongside the main project, and using the “Macro” block to embed it within the main project.

Alternatively, you can create projects within the dedicated “Macros” folder, and they will become immediately available to all users and projects in other folders on the server, via the Add Block menu.

Read more:

• Macros in Omniscope Evo
• How to create an Omniscope macro

 

Optimised report mobile experience

Design dazzling dashboards for different devices – preview multiple layouts and optimise every user’s experience.

The report creator can share a link to a dashboard, while making sure that the mobile display will be optimised and look great on any device.

The slider preview device is a switch between different screen sizes, e.g. desktop, tablet or phone, allowing the designer to see what the report will look like on someone’s screen. Their report page is shown within bounds of the chosen target preview device.

This design optimisation process will manage device layouts per tab, so that each tab can have different content, i.e. different set of views. Instances of views added in one of those layouts can also be added in another layout for the different screen size from “Add view” dropdown. A report shown on a big screen may have 10 charts, while a small laptop may show 6, and when the same report is opened on a mobile phone – it may display only 4 charts.

See how to build a report for any device with different layouts

Custom Blocks Library

There is now a better way to manage your own custom blocks – preview, update and delete blocks from a central repository called the Blocks Library. The library is maintained from a single folder on disk and every Custom Block (a subfolder comprising Python or R code and JSON metadata) that is added to this folder is available immediately from the block picker. When library blocks are dragged from the block picker to the workflow, they are linked with the Custom Block source file on disk. Any change to the source file is immediately propagated to all linked blocks in all workflows, thus removing the necessity to update them manually.  You can optionally use VCS such as a GitHub working copy for the Blocks Library folder.

Expanding data connectivity and cloud technologies compatibility

If the sky’s the limit and you’re looking to move your reporting to the cloud – you will appreciate the newly-added support for cloud file systems and file types.

We’ve added support for reading and writing to Google Cloud Storage and Amazon S3 buckets, via an option in the ‘File location’ drop-down (in File and File Output blocks), also in the Batch Append and the Append Files blocks.

Another addition is support for reading and writing Apache Avro and Parquet files.

(read here ​​Apache Parquet read/write support)

Added Batch HTTP block: Connect to any data source with the REST API via the Batch HTTP block, added to handle data-driven HTTP requests. Allows you to precisely configure and execute HTTP requests, where everything can be data-driven: custom headers, request/response payload, URL structure, HTTP method.

Bundled Admin dashboards

You will be able to enjoy Omniscope logs data from the comfort of your own admin dashboard – to understand which reports are getting the most hits, pinpoint which scheduled tasks need attention, or monitor the app performance.

Easy data access with instant data visualisation/exploration

Adding ability to drag and drop data files onto the Workflow app and the Projects page. These files are then automatically uploaded, and a file block is created for each uploaded file.

Capture, save and manage fast-flowing data

Help is on the way if you’re dealing with a ‘data firehose’ scenario: the new Data Savepoint block (from the Operations blocks menu) is allowing you to capture and analyse the data whenever you wish. The block enables you to retain execution data indefinitely at any point in your workflow, so you can continue working and building the workflow downstream of a slow and complex upstream workflow.

Savepoint block has Retention option which will set the behaviour to either:

• “Persistent” – which captures data indefinitely for future executions, or
• “Temporary” – which captures data temporarily only, within a single workflow execution, to ensure reuse of the same data downstream if there are multiple pathways.

 

Report auto-refresh

New file data refresh option enables you to configure your Report to auto-refresh on open (see Report settings > Global tab). The first person (in a configurable time period) to open this report will trigger the workflow to execute and extract the latest data from the source, and update the visualisation. Useful for infrequently opened projects, or those without a Scheduler-enabled license (a Scheduler is needed if you wish to avoid your users having to wait for the data refresh to complete).

Diagnose query

Report creators will appreciate that in a report with multiple queries they will be able to quickly diagnose which filter combinations are used for each of their queries, and the impact of a complex combination of move/keep actions and filters.

(see Report settings and View data source section).

Dynamic ranking

Calculate your TopN on the fly (based on filtered data): New blocks on the Report data sources menu – Sort and Limit will allow you to display a ready-made queried, calculated, aggregated and ranked dataset in any view. If you wish to dynamically rank top 10 performers, while allowing the report viewer to exclude some data, then you will want to try this mechanism in your reports.

In the screenshot below, our original dataset with ‘all shipment transactions’ has multiple operations applied – the Query block will enable filtering (on the original dataset ‘All shipments’) and control the flow of data into the Aggregation block, where the transactions’ values are summed, then sorted, and a limit is applied to determine the cut-off point for the resulting dataset (to show just Top10 shipments). Every view in this report will have access to this newly-created dynamic dataset.

Chart data download

When making a chart selection, a details button shows in the main toolbar. When clicked – a table showing the selected records is displayed, with options to copy/download the data.

There is also an option to preview the data in each view, showing a table with the displayed and source data and options to copy / export the data.

 

Commercial plan flexibility

• Adding Team plan licensing support for distinct fixed numbers of Viewers vs Editors concurrent seats.
• New option to create a limit of one session per user. If ticked, only one browser session at a time is allowed for each authenticated user. If the licence has limited seats, this will ensure one user cannot consume multiple seats using different devices, and it will allow installations to support named seats in addition to the default shared seats setup.
  

 

General & admin

New ability to restrict what resources projects can access on the file system. Read more.

• Better disk and memory management, admin UI options, diagnostics and disk cleanup for the data engine
• Better logging and improved Workflow Jobs Queue admin page, to show the reason jobs were started and cancelled, and when they were created;
• Added option in the admin interface to specify additional arguments to Pip, the Python Package Installer, so that proxies, features, certificates, etc. can be defined.
• Updated Log4J version to 2.17.1, for peace of mind. This update addresses various security vulnerability concerns present in older versions (even though in practical terms they were not vulnerabilities in Omniscope due to the restricted manner in which Omniscope uses Log4J)..
• Omniscope now records resource monitor data every 60 seconds to a new log file “resource_monitor.csv”. Previously available only for the last 7 days via an Admin page, it is now persisted indefinitely as a rolling logfile. Includes information such as system CPU load, memory use, swap, process heap and non-heap, open file handles, and free disk space. Useful when correlating system load with Omniscope activity.
• More bundled demos – find them on the Welcome page; on upgraded existing installations, install new/updated demos from Admin. Alternatively – visit our refreshed  demo gallery

 

Workflow

• Showing ‘Last Executed’ time when hovering on blocks and on the Report ‘Refresh data’ button to show when the report data was last updated.
• Bookmarks: Ability to create private block bookmarks (available from create bookmark dialog). Private bookmarks will only be available to the user who has created them.
• Added option to download an ioz file without the report data
• Improving performance of large projects with complex workflows
• Updated Google Campaign manager to v3.5.
• Data Table block: option for in-memory (non-persistent) storage
• Field Filter: A new “Auto-populate” button appears when there are no filter rules configured. It inserts a rule for every existing field in the input, letting you easily customise the rules and protect against future field changes or additions.
• File block (and other blocks that let you browse for a file in Omniscope): Added additional sortable columns “Modified”, “Size” and “Type” to the File Browser.
• File and File Output blocks: read/write remote files over SSH (SFTP, SCP) using an optional private key
• File and Append Files blocks: Added options to retry block execution on I/O error
• Field Organiser: Improving performance, including optimising date format conversions. Easy date formats, via a drop-down menu of common formats, while allowing you to continue entering custom formats.
• Promoting workflow execution stats recording to a non-labs feature. A few rock builds ago, we introduced a labs feature that logs workflow-wide and per-block execution time statistics to a CSV logfile, allowing you to analyse bottlenecks in your workflows. It’s now proven for production use with negligible overhead, and has graduated to non-labs.

 

Reports

• Being able to hide report tabs in the Present mode.
• Supporting 3rd party IFRAME contexts for SSO and Unlimited Viewers licenses, when accessed via HTTPS. However, note that 3rd party IFRAME contexts with single-sign-on (OpenID Connect) are “on notice”, with modern browsers phasing this out over the next 2 years; you should instead ensure your IFRAME embeds are seen as a “same site” as a long-term architecture.
• New options for the spacing/ padding/ border options and behaviour related to creating dividers around the charts

 

Visualisations

Adding option to show the colour key at the top or bottom of the view, with configurable number of lines for the colour key text.

Bar view:

• Improving multi-tier split labels (>1 split)
• Adding option to set Bar split min/max values if numeric or date continuous axis.
• Adding option to always show stack labels, even if the stack is very small.
• Improving performance of Bar queries when stacking and colouring by stack.

 

Scatter view:

• Added option to create reference lines in both X and Y axes.
• Added new ‘Jitter’ option – moves markers by a small random amount to provide a better visual sense of the number of observations when either the X or the Y axis are discrete.

 

Table view

• Added option to limit editing to pre-configured values. A dropdown, rather than a text field, is used to edit the values. Read more here.
• New option to disable editing of number/date fields
• New option to style cells and headers differently for each column

 

Map view: Added UK NUTS levels 1-3 GeoJSON base layer maps

Sankey view: new option to show percentages

Timeline and Item views are now non-experimental 

New experimental view: Bubble view

 

💥 and that’s all folks!  Your feedback, as always, is very appreciated, and we are here to support you. 🙏

#TeamVisokio