Watch a brief video showing some of the highlights.
Send us your questions and feedback – let us know what you think!  (here is a Survey link)

Workflow: improved connectivity and data prep

You can now seamlessly integrate data from a wider range of sources into your workflows – our latest update includes support for the following data sources (and outputs):

• Google Cloud Storage, Amazon S3, Microsoft Azure Data Lake Storage Gen2 in the File Input and File Output blocks.
• Outlook PST files in the File block – users can now open, load email, messages, calendar events and other items.
• Google Drive in the File Input, File Output and Batch Append Folder blocks; support for shared drives in Google Drive file input and output.
• SalesForce connector is updated to the latest API version
• New experimental Google Sheets output block.
• New preparation block “Fill Down” replaces nulls in a field with the last non-null preceding value in the same field.
• The new Parse text block will help you parse the contents of a text field, containing JSON or XML, and output one or more records resulting from each cell. (Read more)
  
• Community Custom Scripts are folded into the main block picker categories such as Inputs, Connectors, Preparation and Analytics. The Community Blocks (and also the Library Blocks) now keep their category’s colour on the workflow.
  In the Admin section there is also an option to hide the Community blocks, if they are not relevant for the particular use case.
• For our users in Asia we’ve improved the “Simplified Chinese” translation, so they can interact with the translated menus and application components.
  We also support custom translations – you can find more information below and let us know if you need help to implement.
  (See article)

You can now pin the workflow blocks palette as a left collapsible panel, which makes it easier for you to locate and apply the desired blocks. This eliminates the need to repeatedly click on the Add Block menu. Keeping the sidebar menu readily accessible will save you time and effort when building complex workflows.

Managing date patterns has never been easier thanks to our latest update, which introduces new date format dropdowns in both the Field Organiser and Report field settings. With these drop-downs, you can quickly search for a specific pattern or add a new one to the list.

Drop files anywhere: you can simply drag a source file from your computer and drop it anywhere in the Omniscope interface, whether it’s on the projects page, within a workflow, or on a File source block. This streamlined process means you can easily upload new data without the need to browse to the file location.

Editing data

Table view’s ‘data edit mode’ turns the table into a rich editing experience, similar to a spreadsheet; with multi-cell selection, copy/paste, inserting/deleting records/fields, renaming fields, resizing columns, also cell edits with an undo/redo option.
Note: Requires the Report input to be a Data Table or Text Input block.

• Refill (only) action in the Data Table block: these blocks can now be “refilled” without refreshing the upstream workflow, by selecting just that block and clicking the Execute button with the Refill icon.
• Text Input block has a new UI making it easier to input and edit data.
• Editing dates now shows a date picker, while the Toolbar button is added to clear sorts.
• There is also a new edit behaviour option to disable/show toggle or always enable editing. Editing can be enabled for some columns and disabled for others (from the Report> Fields settings or the three dot menu on the Report’s Data tab).
• Read more here

New data visualisations and Report UI improvements

New charts on the block: Dumbbell and Lollipop are commonly used to represent changes in data over time or between different groups. Dumbbell charts are particularly useful for showing the difference or change between two data points. They consist of a pair of markers or dots connected by a line, with the length of the line representing the difference between the two data points. They are often used to compare before-and-after data, or to show the difference between two groups of data.  Lollipop charts, on the other hand, are useful for displaying individual data points or values within a dataset. They consist of a vertical line or “lollipop stick” with a circular marker at the end representing the data point. They are particularly effective for highlighting outliers or individual data points that are significantly different from the rest of the data.

• Improvements in Bar, Line and Candlestick charts – addition of technical indicators : Simple Moving Average, Exponential Moving Average and Bollinger Bands. These indicators are dynamic and will instantly be recalculated following filtering and selection.
• Multi-point tooltip support in Line, Area, Radar and Parallel Coordinate views (e.g. the user can display the state of multiple KPIs on a given day).
  
• Added option to reverse the split axis in the Bar, Line and Streamgraph views (e.g. the timeline on the X axis could be showing descending order).
• Scatter view quadrants: option to move X/Y axis to origin.
• Added vertical reference lines in the Line view.
  
• Pivot view improvements: support for copying cell data; showing gridlines in the corner of the Pivot view; plus an option to show column or row titles /headers.
• New “Custom HTML” custom view is available now in all app versions. It allows the user to embed either a static HTML snippet or to dynamically display HTML retrieved from the view input data. It can be used to embed HTML widgets (e.g. a Discord channel widget) or to display HTML content based on the filtering / selection happening in other views. You can see a demo on our sandbox.
  
  
  We have introduced a new navigation UI for the Explore and Design dashboard editing modes, to provide a clearer distinction between editing and viewing reports in Omniscope. The Explore mode allows you to quickly and easily interact with all views on the page, simultaneously exploring data and adding new charts. The Design mode, on the other hand, provides more advanced editing capabilities, with the data/settings sidebars and a more focused single chart edit mode.

We have made improvements to our tabs management system with:
– addition of visible vs hidden tabs
– tab layout : drop-down picker Vs visible tabs navigation
– new tab positioning options: on top/ right/ bottom/ left of the report page
– the tabs can be sorted using drag and drop

Report templates, automated diagnostics

Report Templates are pre-configured projects available from the Projects page, some of which contain stunning visualisations that can be simply dragged onto the workflow and used immediately. You can simply re-configure the source to connect to your account/credentials, check the block search criteria and reload the data. The template project contains a dashboard, which will be ready to go the moment you load the new data!

Diagnose fields dialog – We have introduced a new dialog that automatically appears when there are issues with the data sources, fields, or formulas in any of the views within the report. In cases where the data flowing into the report has changed, the dialog offers help with remapping fields to ensure that your data is correctly integrated into the report. This makes it easier to bookmark, reuse or share report templates.

Although the data is the most important element of your report, appearances can also make a significant impact, so why not take a few moments to polish your presentation? With the Report Styles section, you can transform your report with just two clicks. You now have the option to choose from a selection of pre-configured style Presets, or even create and share your own presets as report styles in XML format. This is a convenient way to preserve combinations of colours, fonts, backgrounds, and borders.

Report layouts: Perfect chart proportions are achieved easily in the Free form report editing mode. The new snap report views feature is a user-friendly way to manipulate the charts using the guidelines and create slick dashboard layouts, including the exposed pixel and % measurements . If you haven’t used the dashboard device optimisation feature – it is worth having a go… Read more

Projects page – welcome tour and banners

We wanted to revisit and improve the user experience – whether the user is an experienced analyst or a total novice. The app is more welcoming and offers help to discover the features, while at the same time facilitates a more intuitive interaction with different interfaces.

Our new users will appreciate the Omniscope application tour popup – a dismissible sequence of hints to guide them through the basics on the Welcome screen, in the workflow app and on the report pages. Not a paperclip in sight!

Even the Projects list page got a bit of a makeover, with the ability to add banners with text, images, gifs and links using Markdown syntax.
Simply add a banner.md file to the projects folder. You will be able to add instructions or display a welcome message for the report consumers when they find themselves facing a long report list…
Read more

Report publishing and Automation

•  “Chain repetitions counter” is a new scheduler feature which allows the user to repeat the action(s) a set number of times until finished, or a failure occurs.
• Dashboards can be saved as .IOR files and publishing can be automated with the new scheduler task “Create IOR file from Report”. Read more
• New chart options from three dot menu – report editors can choose to expose the chart’s underlying data to the viewer (and the source data) and make it downloadable, or decide to hide it (through the “Hide source data” option in Report sharing settings).
• Omniscope views can now be saved and downloaded as images from the same chart menu (in PNG format).
• Convert and publish Omniscope multi-tab report as PowerPoint presentation via the new custom Report to PowerPoint block (available on Business/Enterprise plans).

Advanced features

Report variable influencing a live SQL query. To use this feature, the database block must be configured for live query and you must tick “Enable live query injection” in the report variable configuration. This will allow you to use the variable name as a parameter in the Database block custom SQL option. See the article 

Omniscope File Drop facility: A new Custom view allows the report viewer to upload a data file by uploading it to a Report page and immediately trigger a data Workflow execution of specific blocks in one request (where the uploaded data file will become an input data block), optionally also refreshing the report which contains the ‘upload file’ view. See more

See also – relevant features already available:

Parameter passing – Custom views in the Report section allow the analyst to integrate a parameter-passing mechanism, so that the report viewer is able to make a choice or enter a new value to be passed to the workflow and trigger a re-execution.
See more

Performance improvements

When multiple users are requesting the same query Omniscope has an option to store and retrieve results of the first-executed query. Report queries are cached indefinitely, subject to a configurable cache memory capacity via an option in the Admin section, and do not otherwise expire until the server app is shut down. Manual report “warm-up” (after data refresh) can be useful on servers with complex reports and large numbers of viewers to facilitate dramatically improved load time and interaction time for common initial actions.

Important Note about the Omniscope Classic Mobile mode sunset – for the users who wish to share their reports as web links Evo is a superior web app – it is better positioned to deliver web based reports and share content. As of this Rock build we will remove support for the old Classic ‘mobile mode’. Please get in touch if you wish to discuss the migration options and licensing.

Let us know what you think, here is a Survey link

#TeamVisokio

The post Omniscope Rock Build – 2023.1 appeared first on Visokio.

No matter the topic or the vertical, once you know where is your data, in any form or shape, Omniscope will provide you with a unified data platform and a simple UI to build and deploy customised web apps for data processing, analytics and reporting. And you don’t need to know how to code.

You may have some questions, as starter “how do I access and load data“? Well, Omniscope has enough core workflow blocks to access files, web resources, databases, services, in various formats, from flat files to structured csv, Excel, JSON and XML. You just need to add and configure a block, once, and re-use it whenever you need to.

At this point you could ask: “I’ve got the data, but how do I clean it, see? It’s all messed up.” Well (2), you’ll find a rich data preparation section to solve the usual problems,  like duplicated records, misspelt words, missing values, and to perform typical operations like pivot, transpose, join, append, aggregate, sort, filter and validate data. You will have enough routines for data prep and ETL, to build workflows to load, stream, blend and transform any data.

“But what about data analytics and machine learning, AI?” In the Analytics section we’ve bundled quite a few outstanding blocks, to deal with the most common data problems solved with classic math, statistics and machine learning algorithms. You will be able (point and click) to perform regression to provide causal analysis or to predict the value of missing data. You’ll find groups (clustering), find associations between them (basket analysis) and anomalies in your data. And more, like leveraging time series forecast, network analysis, text analysis and topic modelling. I am sure you will have enough no-code-data-weapons to extract insights.

“Sure, but how do I build the app UX and UI to share my findings?”
Dataviz? All drag-and-drop, from a set of predefined charts and views, you will be able to build reports, dashboards, or simply interactive web content / pages / tabs, allowing your end user to navigate and interact with graphs, select and filter data, drilling down, and even editing and inputting data if you wanted. You will build the final UX for your end users and embed the customised UI on your site or, viceversa, embed any other web component into the Omniscope “report” single page web app.

“Cool, but does it scale?” Data volume should not be a problem. If you need to process big data, data preparation will leverage all your machine / server resources, in a batch streaming fashion where possible. And if your data is on a SQL database, no problem, you will be able to directly live query data, and leverage your cloud SQL DB to power your dashboards.

“OK… what about automation?” It’s the key, right? Once you build your workflows, web reports and dashboards you want to make sure everything comes to life, showing the recent version of data, and reacting on events. The Scheduler app and the Automation API will be there to run parameterised data refresh & report updates, to trigger tasks, data alerts, and configure, edit & query data. You’ll have all the tools to have Omniscope talk to other software or services, and become your friendly robot. You’ll instruct it once, and it’ll do the job for you. while you can use your time wisely to invent more.

“Sounds interesting, so…. where do I run it, do I need a particular machine or device?”
Omniscope is a web app. It was designed to be universal and deliver a fresh and smart experience on any device: Windows, Mac, Linux, Android, iOS, you just need a web browser, simple (…though better if recent Chrome, Firefox and Edge).
And regarding deployment and security, don’t worry, we won’t see your data. You can install Omniscope on-premises or on your private cloud (AWS, Google Cloud or Azure), and control data and user access / permissions using modern protocols like OIDC and SSO.

The final touch, before you ask: it’s a white-label software: you can host branded data solutions and embedded analytics, get rid of that Visokio Omniscope logo and put your supa-dupa company branding in place. We don’t care about our branding, we only care about building and giving you the best tool for your business.

Ah, and of course, if you are a developer or you’ve got some magic code, be my guest! We made sure you can extend Omniscope and build your own data blocks and custom UI, to augment the data pipelines with your Python / R data transformation and ML/AI scripts , and enhance the web UI with any HTML+JS web components and/or charts.
So you can really achieve the “omni scope” and deliver custom branded data solutions bundling your secret sauce.

Does all this sound like what you’re looking for? Drop us a line, check some demos or give it a go on our public sandbox.

Peace out!

The post No-code Development Platform to build data web apps appeared first on Visokio.

Bring in the Robots!

Save time with the Macro functionality – create reusable workflow logic, then add it as an ETL routine to other projects. 

The new Rock build adds the ability to decompose complex workflows into smaller “macros” and optionally share them with other users/ folders/ projects on the server via the Macros folder.

You can use macros without sharing them, as a way of reducing the complexity of a large workflow, by creating a smaller project alongside the main project, and using the “Macro” block to embed it within the main project.

Alternatively, you can create projects within the dedicated “Macros” folder, and they will become immediately available to all users and projects in other folders on the server, via the Add Block menu.

Read more:

• Macros in Omniscope Evo
• How to create an Omniscope macro

Optimised report mobile experience

Design dazzling dashboards for different devices – preview multiple layouts and optimise every user’s experience.

The report creator can share a link to a dashboard, while making sure that the mobile display will be optimised and look great on any device.

The slider preview device is a switch between different screen sizes, e.g. desktop, tablet or phone, allowing the designer to see what the report will look like on someone’s screen. Their report page is shown within bounds of the chosen target preview device.

This design optimisation process will manage device layouts per tab, so that each tab can have different content, i.e. different set of views. Instances of views added in one of those layouts can also be added in another layout for the different screen size from “Add view” dropdown. A report shown on a big screen may have 10 charts, while a small laptop may show 6, and when the same report is opened on a mobile phone – it may display only 4 charts.

See how to build a report for any device with different layouts

Custom Blocks Library

There is now a better way to manage your own custom blocks – preview, update and delete blocks from a central repository called the Blocks Library. The library is maintained from a single folder on disk and every Custom Block (a subfolder comprising Python or R code and JSON metadata) that is added to this folder is available immediately from the block picker. When library blocks are dragged from the block picker to the workflow, they are linked with the Custom Block source file on disk. Any change to the source file is immediately propagated to all linked blocks in all workflows, thus removing the necessity to update them manually.  You can optionally use VCS such as a GitHub working copy for the Blocks Library folder.

Expanding data connectivity and cloud technologies compatibility

If the sky’s the limit and you’re looking to move your reporting to the cloud – you will appreciate the newly-added support for cloud file systems and file types.

We’ve added support for reading and writing to Google Cloud Storage and Amazon S3 buckets, via an option in the ‘File location’ drop-down (in File and File Output blocks), also in the Batch Append and the Append Files blocks.

Another addition is support for reading and writing Apache Avro and Parquet files.

(read here ​​Apache Parquet read/write support)

Added Batch HTTP block: Connect to any data source with the REST API via the Batch HTTP block, added to handle data-driven HTTP requests. Allows you to precisely configure and execute HTTP requests, where everything can be data-driven: custom headers, request/response payload, URL structure, HTTP method.

Bundled Admin dashboards

You will be able to enjoy Omniscope logs data from the comfort of your own admin dashboard – to understand which reports are getting the most hits, pinpoint which scheduled tasks need attention, or monitor the app performance.

Easy data access with instant data visualisation/exploration

Adding ability to drag and drop data files onto the Workflow app and the Projects page. These files are then automatically uploaded, and a file block is created for each uploaded file.

Capture, save and manage fast-flowing data

Help is on the way if you’re dealing with a ‘data firehose’ scenario: the new Data Savepoint block (from the Operations blocks menu) is allowing you to capture and analyse the data whenever you wish. The block enables you to retain execution data indefinitely at any point in your workflow, so you can continue working and building the workflow downstream of a slow and complex upstream workflow.

Savepoint block has Retention option which will set the behaviour to either:

• “Persistent” – which captures data indefinitely for future executions, or
• “Temporary” – which captures data temporarily only, within a single workflow execution, to ensure reuse of the same data downstream if there are multiple pathways.

Report auto-refresh

New file data refresh option enables you to configure your Report to auto-refresh on open (see Report settings > Global tab). The first person (in a configurable time period) to open this report will trigger the workflow to execute and extract the latest data from the source, and update the visualisation. Useful for infrequently opened projects, or those without a Scheduler-enabled license (a Scheduler is needed if you wish to avoid your users having to wait for the data refresh to complete).

Diagnose query

Report creators will appreciate that in a report with multiple queries they will be able to quickly diagnose which filter combinations are used for each of their queries, and the impact of a complex combination of move/keep actions and filters.

(see Report settings and View data source section).

Dynamic ranking

Calculate your TopN on the fly (based on filtered data): New blocks on the Report data sources menu – Sort and Limit will allow you to display a ready-made queried, calculated, aggregated and ranked dataset in any view. If you wish to dynamically rank top 10 performers, while allowing the report viewer to exclude some data, then you will want to try this mechanism in your reports.

In the screenshot below, our original dataset with ‘all shipment transactions’ has multiple operations applied – the Query block will enable filtering (on the original dataset ‘All shipments’) and control the flow of data into the Aggregation block, where the transactions’ values are summed, then sorted, and a limit is applied to determine the cut-off point for the resulting dataset (to show just Top10 shipments). Every view in this report will have access to this newly-created dynamic dataset.

Chart data download

When making a chart selection, a details button shows in the main toolbar. When clicked – a table showing the selected records is displayed, with options to copy/download the data.

There is also an option to preview the data in each view, showing a table with the displayed and source data and options to copy / export the data.

Commercial plan flexibility

• Adding Team plan licensing support for distinct fixed numbers of Viewers vs Editors concurrent seats.
• New option to create a limit of one session per user. If ticked, only one browser session at a time is allowed for each authenticated user. If the licence has limited seats, this will ensure one user cannot consume multiple seats using different devices, and it will allow installations to support named seats in addition to the default shared seats setup.
  

General & admin

New ability to restrict what resources projects can access on the file system. Read more.

• Better disk and memory management, admin UI options, diagnostics and disk cleanup for the data engine
• Better logging and improved Workflow Jobs Queue admin page, to show the reason jobs were started and cancelled, and when they were created;
• Added option in the admin interface to specify additional arguments to Pip, the Python Package Installer, so that proxies, features, certificates, etc. can be defined.
• Updated Log4J version to 2.17.1, for peace of mind. This update addresses various security vulnerability concerns present in older versions (even though in practical terms they were not vulnerabilities in Omniscope due to the restricted manner in which Omniscope uses Log4J)..
• Omniscope now records resource monitor data every 60 seconds to a new log file “resource_monitor.csv”. Previously available only for the last 7 days via an Admin page, it is now persisted indefinitely as a rolling logfile. Includes information such as system CPU load, memory use, swap, process heap and non-heap, open file handles, and free disk space. Useful when correlating system load with Omniscope activity.
• More bundled demos – find them on the Welcome page; on upgraded existing installations, install new/updated demos from Admin. Alternatively – visit our refreshed  demo gallery

Workflow

• Showing ‘Last Executed’ time when hovering on blocks and on the Report ‘Refresh data’ button to show when the report data was last updated.
• Bookmarks: Ability to create private block bookmarks (available from create bookmark dialog). Private bookmarks will only be available to the user who has created them.
• Added option to download an ioz file without the report data
• Improving performance of large projects with complex workflows
• Updated Google Campaign manager to v3.5.
• Data Table block: option for in-memory (non-persistent) storage
• Field Filter: A new “Auto-populate” button appears when there are no filter rules configured. It inserts a rule for every existing field in the input, letting you easily customise the rules and protect against future field changes or additions.
• File block (and other blocks that let you browse for a file in Omniscope): Added additional sortable columns “Modified”, “Size” and “Type” to the File Browser.
• File and File Output blocks: read/write remote files over SSH (SFTP, SCP) using an optional private key
• File and Append Files blocks: Added options to retry block execution on I/O error
• Field Organiser: Improving performance, including optimising date format conversions. Easy date formats, via a drop-down menu of common formats, while allowing you to continue entering custom formats.
• Promoting workflow execution stats recording to a non-labs feature. A few rock builds ago, we introduced a labs feature that logs workflow-wide and per-block execution time statistics to a CSV logfile, allowing you to analyse bottlenecks in your workflows. It’s now proven for production use with negligible overhead, and has graduated to non-labs.

Reports

• Being able to hide report tabs in the Present mode.
• Supporting 3rd party IFRAME contexts for SSO and Unlimited Viewers licenses, when accessed via HTTPS. However, note that 3rd party IFRAME contexts with single-sign-on (OpenID Connect) are “on notice”, with modern browsers phasing this out over the next 2 years; you should instead ensure your IFRAME embeds are seen as a “same site” as a long-term architecture.
• New options for the spacing/ padding/ border options and behaviour related to creating dividers around the charts

Visualisations

Adding option to show the colour key at the top or bottom of the view, with configurable number of lines for the colour key text.

Bar view:

• Improving multi-tier split labels (>1 split)
• Adding option to set Bar split min/max values if numeric or date continuous axis.
• Adding option to always show stack labels, even if the stack is very small.
• Improving performance of Bar queries when stacking and colouring by stack.

Scatter view:

• Added option to create reference lines in both X and Y axes.
• Added new ‘Jitter’ option – moves markers by a small random amount to provide a better visual sense of the number of observations when either the X or the Y axis are discrete.

Table view

• Added option to limit editing to pre-configured values. A dropdown, rather than a text field, is used to edit the values. Read more here.
• New option to disable editing of number/date fields
• New option to style cells and headers differently for each column

Map view: Added UK NUTS levels 1-3 GeoJSON base layer maps

Sankey view: new option to show percentages

Timeline and Item views are now non-experimental 

New experimental view: Bubble view

and that’s all folks!  Your feedback, as always, is very appreciated, and we are here to support you.

#TeamVisokio

The post Omniscope Rock Build – 2022.1 appeared first on Visokio.

The first step in the process is thinking about meaningful questions, so a report can be developed to deliver relevant answers.

With the world’s attention span getting shorter, the analyst will also want to deliver these with a minimal number of charts. This is where the data artisan skills become useful…

Open data sources are everywhere and, in many cases, they come in a format that is not accessible to all potential users (xml, JSON, not just Excel), which means that only patient/ data literate users can interrogate the information and extract meaningful insights.

Curious? This tutorial will show you how to build the interactive dashboard presented below. Read more…

Feel free to play with the charts and filters to see the effects!

The post How to build a live data transaction report appeared first on Visokio.

But how about security? How to run a script in a isolated sandbox so you can be sure it’s not going to harm your machine, service or environment.

A while back we’ve introduced the concept of Custom Blocks in Omniscope. Basically anyone can build their own block by using Python or R to expand the set of blocks we bundle and to enable our users to augment their workflow with any scripts, from new analytics algorithms to simple data transformation routines.
However, such scripts have unrestricted access to the filesystem… so you would have to trust your workflow-permissioned users and the designers who create those blocks.
We’ve then decided to make our Omniscope users life easier, and safer!
Omniscope 2021.2 lets you execute all your custom blocks in a Docker container, via a server-wide policy, where the script code is completely isolated from the server – and its file system other than where configured via options (to mount and map single files or folders).

Set up Docker Desktop (Windows/Mac) or Docker daemon (Linux) and you are ready to go. Read more here

Want to try out? Feel free to use our public sandbox on http://daily.omniscope.me . Happy and safe coding!

The post Secure Python / R scripts execution in Docker with Omniscope appeared first on Visokio.

The post Omniscope Evo: 15 seconds explainer video appeared first on Visokio.

New look welcome page and project templates

The Omniscope welcome page has changed! As well as finding your projects and data files, you can now create projects for common tasks using templates, such as for importing from databases or entering/editing data. Recent files are shown separately, so you can quickly find the project you were working on yesterday. The interface is now more consistent and familiar across the welcome page, workflow and report experiences.

Grouping blocks

Tired of zooming and scrolling through large, complicated workflows? Lost in a sea of blocks and connections? Now you can organise sections of your workflow into groups, optionally name them, and expand them with a click when you need to look or work inside.

Animated transitions in reports

Your reports will come to life when your users filter and select within them, now that Omniscope animates most transitions.
As well as being more funky and engaging, it helps perceive the impact of actions upon the data.

New views

Need to see the cumulative effect of a bar chart? The new Waterfall chart lets you do this, with an optional total bar, and with negative (down) as well as positive (up) sections. Also known as cascading bars, flying bricks or bridge.

Still experimental in this rock, and due to graduate early in version 2021.3, are the new Timeline and Item views. Use the Timeline view to show a small number of events on a single time axis, coloured and labelled. Use the Item view to represent a small number of “items” (e.g. seats in parliament) as coloured dots in one of 4 common arrangements.

Workflow performance 

Performance interacting with and executing workflows has been greatly improved. Memory use under high load is better managed. Experimental “labs” options have been added to let you try other performance improvements in development (please send feedback). Should things go wrong, improved diagnostics will help us understand and resolve the problem faster.

Upgraded data engine

The bundled data engine, used to execute queries underlying the visualisations in your reports, has been upgraded, bringing performance, reliability and disk management improvements. Existing servers will need to rebuild their queryable data from execution data (automatically on first access).
You can find documentation and instructions here.

Secure, isolated Custom Block execution using Docker

Custom Blocks let you write your own code to transform and process data, in R or Python, and then share them as bookmarks or community blocks on GitHub. However, such scripts have unrestricted access to the server filesystem, previously requiring you to trust your workflow-permissioned users. Omniscope 2021.2 now lets you execute all custom blocks using Docker, via a server-wide policy, where the script code is completely isolated from the server file system other than where configured via options. Requires Docker Desktop (Windows/Mac) or Docker daemon (Linux) to be installed and running. (read more)

…and more:

Data preparation

• Templated parameters – new powerful type of param that helps with URL / file path customisation (read more)
• The Google Sheets connector lets you pull data natively from one of your cloud spreadsheets
• Native support for more database vendors. Although Omniscope supports any JDBC based connector, there are now additional guided choices for SQLite, MariaDB, Azure SQL Server, Amazon Redshift, Snowflake, Vertica and Actian Avalanche.
• The Database Output block lets you customise the schema on a per-field basis when creating tables during execution.
• In Field Organiser, the new NETWORKDAYS formula function lets you easily calculate working days between two dates.
• If you’re a DoubleClick user, please note, it has been renamed to Campaign Manager.
• Community blocks:
  • This month’s hot giveaway – new Community block: Field Renamer – just feed it a mapping file containing Old Vs New field names (and your dataset) and it will rename the fields. Faster than typing 100s of new names in the Field Organiser block!

• • Slack API web client: it allows you to call all the public Slack endpoints and methods.
  • Azure Data Lake Storage Gen2 Blob Connector

Interface 

• You can now add notes to every block in the workflow. Useful if you want to explain what you were doing in that 5th Field Organiser block!
• Report tabs can now actually be shown as … tabs! We’ve kept the default (a drop-down) for space efficiency, but you can now enable actual tabs (as a separate toolbar, or configured to appear on the main toolbar if space permits).
• In report, the new Styles menu lets you quickly and easily make common style changes to your views, such as setting the font size.
• Sharing links are now accompanied by QR codes. Useful on a large screen or at a show, and when testing on mobile devices.
• IE, Legacy Edge and Opera mini are no longer supported.

DataViz

• Filter devices have better defaults. Choice devices will be the default, where the data is categorical.
• The Map view improvements: you can show arrows on your “Line” layers – useful to show transport flows, for example. A new Layer picker overlay lets your report users choose what layers they see. And you can preview what’s inside GeoJSON files more easily – useful when configuring how the additional data is rendered and merged.
• Bar, Line and other views now let you make drag-rectangle selections to rapidly and easily select multiple bars or lines.
• Many views now let you define custom tooltips in a consistent and powerful way, so you can manage what information is displayed on mouse-over in visualisations.
• Custom Views: “Set parameters and execute” view – it creates an interface where the report viewer can set the new parameter value and trigger a workflow execution

Admin, branding, permissions

• Manual activation in Evo – relevant in cases where ‘online’ activation/deactivation is blocked by the company’s firewall.
• Improved Admin / Scheduler with multiple working on same tasks/settings.
• New scheduler actions: Refresh all Report blocks and Publish all output blocks.
• Fix for Mac activation lost on system OSX software update.
• Fully brandable welcome/project list.
• Permission presets: Simplification of applying permissions. Ability to define globally (Admin section) permission presets which you can then choose from in permissions dialog.

As always, your feedback is highly appreciated, and we are here for you – if you need help, advice or training to put you in a better shape to deal with data challenges.

Let us be your Q!

Love from #TeamVisokio

The post Omniscope Rock build – 2021.2 appeared first on Visokio.

The Slack API WebClient connector allows you to execute any permissioned Slack API method, meaning you can also use it to set options, join channels, kick users, post messages on channels and automate any action in a data-driven way.

Got Omniscope Evo handy? Download the projects to experiment at your end.

The post How to import any data from a Slack workspace and analyse it in Omniscope Evo appeared first on Visokio.

Download the latest rock build here.

Explore mode

Surf your reports in the new Explore mode – interact with data and chart configuration simultaneously and enjoy the immediate effects. Try all four modes on the main toolbar: Explore (for free-form data discovery), Design (for authoring pixel-perfect dashboards), Present (preview the report as your users would see it) and Focus (zoom into a single view, with drag-and-drop fields, measures and dimensions).

We’ve also comprehensively reorganised view options into meaningful sections  such as “Measure axis”, “Lines”, “Bars”, “Markers”, with a cog menu that lets you search everything. 

Improved Authentication – OpenID Connect

Check out the improved authentication methods – support for session-based authentication using OpenID Connect identity providers (Google, Azure, Keycloak, Okta, Auth0) – suite of configurations supporting flexible security models, role-based authentication managed at the provider, identity management and tracking, account switching, single-sign on (SSO), session expiry and refresh, multi-tenant hosting.

The Admin and Scheduler parts of the app are now part of the same authentication scope, with new permissions available at the root level to control access to them.

New “super-groups”, let you define a server-wide or folder-wide configuration of (e.g.) administrator accounts, which cannot be overridden in subfolders. Existing server deployments will find that the admin credentials have automatically been migrated to an equivalent super-group.

Documentation, as usual, available on our Help Center Solutions section.

Publishing on steroids

Publishing 1 report with 100 variants? No worries – we’ve got your back. Report Generator block will automate creation of reports from a template, including the custom tabs/filters driven by a data configuration (different from the multitenant system).

Give your report viewer more than just a variant – add a Refresh Data button, enabling them to retrieve the latest information at any time. 

Integrate email into your reporting process – either for email data harvesting using the new Email block including the ability to download attachments, or to send out data-triggered alerts (from Validation block) and new report links (via Email output).

Filter, drill down and share your data exploration

Share insights by using the new “Sharable link” option, while preserving the exact tab and filter states. Or build links with customised filters yourself, using our scheme to specify filters. Use the new Filters button to hide and show your dashboard’s filters. Show view menus reveals a menu per view, to access data export and maximise controls.

We brought new blocks and the tools to build your own

Meet the new blocks in town: Storage – use it as an input or output, sharing the data by table name across projects. Previously experimental, this has now graduated. It also now lets you write data in “append mode”, such that every time you refresh it, the input data is appended to instead of replacing the existing data. Project Metadata block: extract metadata about blocks, outputs, schema, reports, tabs, data sources from a project, to facilitate “configuration as data” automation workflows.  Spring clean any workflow with new “Clean-up data” action and delete data not used by reports – available as in the workflow app menu, as a scheduler task, and also automatically in the background for older projects. 

Previously experimental, the Custom block lets you create reusable blocks from your own Python or R code, and share them with your team, or publish them on our Github community blocks repository.  Now supporting R 4.x, with an option to select an R version.

Misc

• The File Output block now has an Append option when writing to IOD files.
• New formula functions SUBSET_UNIQUECOUNT, TEXTVALUE and INTROUND available in reports.
• You can add Reference Lines to Bar and Line views, and have better control over multiple axes.
• The Bar view lets you configure a custom sort measure, supporting formulas.
• The Workflow API has a new Lambda Workflow Execution endpoint, allowing you to spawn and execute parameterised clones of workflows.
• The Omniscope Setup Console makes setting up a headless Linux server a breeze, with guided licence activation, configuration of the “external web server”, and setup of an admin account. With a silent mode for fully automated deployments. See the Linux Installation help article.
• Omniscope now by default runs with restricted filesystem access when running as an external web server (Team license and up). Your workflows and blocks can no longer access files outside the sharing folder by default. You can revert to the old behaviour in environments where this is not a security concern, using “Allow using files outside Sharing Folder” in Admin, web server settings.
• We’ve renamed the launch scripts on Linux to avoid confusion. The old scripts remain until 2021.2, when they’ll be removed.
• We’ve updated the version of Java that Omniscope runs on and bundles – now Java 11. This requires some configuration changes to custom deployment scripts and Windows Service configurations; see advisory.
• After frankly years of suffering as developers, following Microsoft’s lead and schedule, we are finally dropping support for Microsoft Internet Explorer (IE) and Edge Legacy. 2021.1 is the last version to support these browsers, with a warning displayed. 2021.2 will fully drop support. The latest versions of Chrome or new Edge are recommended.
• Memory: some improvements to efficiency, and new diagnostics for use in production servers.
• And as always, a huge number of minor enhancements, fixes, stability and diagnostic improvements. See the full changelog for details.

WHAT’S NEXT? YOUR FEATURES!

As always, we’d love to hear what you want added to Omniscope. Please contact us to get access and join the discussion at our public Trello board!

The post Omniscope ROCK build – 2021.1 appeared first on Visokio.

Download the latest rock build here.

NOTABLE NEW FEATURES:

Easily break down your data, with Hierarchies

Hierarchies let you group and collapse/expand your data by successive levels of detail. Date and number fields have implicit hierarchies (year, quarter, month…), and you can define your own custom hierarchies (product category, sub-category…). When you drag on fields which are part of a hierarchy, a +/- button inside the “pill” lets you easily add/remove further levels. And within your dashboard, +/- buttons let users expand and collapse for themselves.

Recycle, Reuse – save time with Bookmarks

With Bookmarks, Omniscope now lets you easily save a block configuration for reuse elsewhere within the same Omniscope installation. Let’s say you’ve configured the perfect Field Organiser block, entered some tricky connection details or SQL query in a Database source block, developed a Custom block using your own handwritten Python or R, or finished a Report block that needs to become a company dashboard template. Bookmark it, and reuse it from the Add Block menu on another project, available to other users (if it’s a shared server).

Data Table block – edit/snapshot the data in the workflow or report

This new block represents an editable and persistent store of data (it’s simply a table of data, hence the name), large or small. You can put data into it (“refreshing” from the input, if connected, as a persistent snapshot of the incoming data), pull data out of it (by connecting blocks downstream), consume directly as live data in the report (no need to execute), and edit it (in the workflow, directly in your reports, and via APIs, if permissioned and enabled, with immediate effect). See introduction and use with workflow execution API.

Salesforce input and output blocks

We’ve added Salesforce source and output blocks, so you can now consume tables from Salesforce in your Omniscope workflows, enrich and correct the data, then publish back.

Categorical axis in Scatter

You can now create scatter views using categorical (text) fields as either axis.

And more…

As usual, tons of lesser features and fixes have been added since the previous rock build, including:

• Workflow APIs now let you execute and set parameter values
• Much improved performance with large (e.g. 200+ block) workflows
• Deployment on Windows and Linux VMs from all major clouds (MS Azure, GCE, AWS EC2) fully supported (see advisory)
• Easily consume JSON data from REST endpoints using the new “JSON HTTP API” block
• Send emails with data attached using the new Email Output block, and automate sending such emails with batch publishing actions using the Batch Output block.
• Perform better automatic checks on your data, and consume problem metadata, using the latest enhancements to the Validate Data block.
• Improved robustness of Storage Block, and workflow execution in general, in high concurrency servers.

Community blocks

Over the same time period, a number of notable new community blocks have been shared on our Github repository. These are open-source add-ons which use open APIs within Omniscope to add functionality, and can be found within the Add Block menu:

• Sharepoint Online Downloader block
• ForEach block
• Etherscan block
• SFTP Downloader block
• Report to PDF batch output block
• Web Image PDF output block

Advisory notes

Any Omniscope cloud installation (on Microsoft Azure Compute, Google Compute Engine or Amazon EC2) must be deactivated/reactivated before/after upgrading to 2020.3. See details.

What’s next? Your features!

We’re busy developing OpenID Connect integration for better identity and authentication management, expanded formulas, drill-down, dashboard filters as URL parameters, and smarter defaults for filters.

But we’d love to hear what you want added to Omniscope. Please contact us to get access and join the discussion at our new public Trello board!

The post Omniscope ROCK build – 2020.3 appeared first on Visokio.