Watch a brief video showing some of the highlights.
Send us your questions and feedback – let us know what you think!  (here is a Survey link)

Workflow: improved connectivity and data prep

You can now seamlessly integrate data from a wider range of sources into your workflows – our latest update includes support for the following data sources (and outputs):

• Google Cloud Storage, Amazon S3, Microsoft Azure Data Lake Storage Gen2 in the File Input and File Output blocks.
• Outlook PST files in the File block – users can now open, load email, messages, calendar events and other items.
• Google Drive in the File Input, File Output and Batch Append Folder blocks; support for shared drives in Google Drive file input and output.
• SalesForce connector is updated to the latest API version
• New experimental Google Sheets output block.
• New preparation block “Fill Down” replaces nulls in a field with the last non-null preceding value in the same field.
• The new Parse text block will help you parse the contents of a text field, containing JSON or XML, and output one or more records resulting from each cell. (Read more)
  
• Community Custom Scripts are folded into the main block picker categories such as Inputs, Connectors, Preparation and Analytics. The Community Blocks (and also the Library Blocks) now keep their category’s colour on the workflow.
  In the Admin section there is also an option to hide the Community blocks, if they are not relevant for the particular use case.
• For our users in Asia we’ve improved the “Simplified Chinese” translation, so they can interact with the translated menus and application components.
  We also support custom translations – you can find more information below and let us know if you need help to implement.
  (See article)

You can now pin the workflow blocks palette as a left collapsible panel, which makes it easier for you to locate and apply the desired blocks. This eliminates the need to repeatedly click on the Add Block menu. Keeping the sidebar menu readily accessible will save you time and effort when building complex workflows.

Managing date patterns has never been easier thanks to our latest update, which introduces new date format dropdowns in both the Field Organiser and Report field settings. With these drop-downs, you can quickly search for a specific pattern or add a new one to the list.

Drop files anywhere: you can simply drag a source file from your computer and drop it anywhere in the Omniscope interface, whether it’s on the projects page, within a workflow, or on a File source block. This streamlined process means you can easily upload new data without the need to browse to the file location.

Editing data

Table view’s ‘data edit mode’ turns the table into a rich editing experience, similar to a spreadsheet; with multi-cell selection, copy/paste, inserting/deleting records/fields, renaming fields, resizing columns, also cell edits with an undo/redo option.
Note: Requires the Report input to be a Data Table or Text Input block.

• Refill (only) action in the Data Table block: these blocks can now be “refilled” without refreshing the upstream workflow, by selecting just that block and clicking the Execute button with the Refill icon.
• Text Input block has a new UI making it easier to input and edit data.
• Editing dates now shows a date picker, while the Toolbar button is added to clear sorts.
• There is also a new edit behaviour option to disable/show toggle or always enable editing. Editing can be enabled for some columns and disabled for others (from the Report> Fields settings or the three dot menu on the Report’s Data tab).
• Read more here

New data visualisations and Report UI improvements

New charts on the block: Dumbbell and Lollipop are commonly used to represent changes in data over time or between different groups. Dumbbell charts are particularly useful for showing the difference or change between two data points. They consist of a pair of markers or dots connected by a line, with the length of the line representing the difference between the two data points. They are often used to compare before-and-after data, or to show the difference between two groups of data.  Lollipop charts, on the other hand, are useful for displaying individual data points or values within a dataset. They consist of a vertical line or “lollipop stick” with a circular marker at the end representing the data point. They are particularly effective for highlighting outliers or individual data points that are significantly different from the rest of the data.

• Improvements in Bar, Line and Candlestick charts – addition of technical indicators : Simple Moving Average, Exponential Moving Average and Bollinger Bands. These indicators are dynamic and will instantly be recalculated following filtering and selection.
• Multi-point tooltip support in Line, Area, Radar and Parallel Coordinate views (e.g. the user can display the state of multiple KPIs on a given day).
  
• Added option to reverse the split axis in the Bar, Line and Streamgraph views (e.g. the timeline on the X axis could be showing descending order).
• Scatter view quadrants: option to move X/Y axis to origin.
• Added vertical reference lines in the Line view.
  
• Pivot view improvements: support for copying cell data; showing gridlines in the corner of the Pivot view; plus an option to show column or row titles /headers.
• New “Custom HTML” custom view is available now in all app versions. It allows the user to embed either a static HTML snippet or to dynamically display HTML retrieved from the view input data. It can be used to embed HTML widgets (e.g. a Discord channel widget) or to display HTML content based on the filtering / selection happening in other views. You can see a demo on our sandbox.
  
  
  We have introduced a new navigation UI for the Explore and Design dashboard editing modes, to provide a clearer distinction between editing and viewing reports in Omniscope. The Explore mode allows you to quickly and easily interact with all views on the page, simultaneously exploring data and adding new charts. The Design mode, on the other hand, provides more advanced editing capabilities, with the data/settings sidebars and a more focused single chart edit mode.

We have made improvements to our tabs management system with:
– addition of visible vs hidden tabs
– tab layout : drop-down picker Vs visible tabs navigation
– new tab positioning options: on top/ right/ bottom/ left of the report page
– the tabs can be sorted using drag and drop

Report templates, automated diagnostics

Report Templates are pre-configured projects available from the Projects page, some of which contain stunning visualisations that can be simply dragged onto the workflow and used immediately. You can simply re-configure the source to connect to your account/credentials, check the block search criteria and reload the data. The template project contains a dashboard, which will be ready to go the moment you load the new data!

Diagnose fields dialog – We have introduced a new dialog that automatically appears when there are issues with the data sources, fields, or formulas in any of the views within the report. In cases where the data flowing into the report has changed, the dialog offers help with remapping fields to ensure that your data is correctly integrated into the report. This makes it easier to bookmark, reuse or share report templates.

Although the data is the most important element of your report, appearances can also make a significant impact, so why not take a few moments to polish your presentation? With the Report Styles section, you can transform your report with just two clicks. You now have the option to choose from a selection of pre-configured style Presets, or even create and share your own presets as report styles in XML format. This is a convenient way to preserve combinations of colours, fonts, backgrounds, and borders.

Report layouts: Perfect chart proportions are achieved easily in the Free form report editing mode. The new snap report views feature is a user-friendly way to manipulate the charts using the guidelines and create slick dashboard layouts, including the exposed pixel and % measurements . If you haven’t used the dashboard device optimisation feature – it is worth having a go… Read more

Projects page – welcome tour and banners

We wanted to revisit and improve the user experience – whether the user is an experienced analyst or a total novice. The app is more welcoming and offers help to discover the features, while at the same time facilitates a more intuitive interaction with different interfaces.

Our new users will appreciate the Omniscope application tour popup – a dismissible sequence of hints to guide them through the basics on the Welcome screen, in the workflow app and on the report pages. Not a paperclip in sight!

Even the Projects list page got a bit of a makeover, with the ability to add banners with text, images, gifs and links using Markdown syntax.
Simply add a banner.md file to the projects folder. You will be able to add instructions or display a welcome message for the report consumers when they find themselves facing a long report list…
Read more

Report publishing and Automation

•  “Chain repetitions counter” is a new scheduler feature which allows the user to repeat the action(s) a set number of times until finished, or a failure occurs.
• Dashboards can be saved as .IOR files and publishing can be automated with the new scheduler task “Create IOR file from Report”. Read more
• New chart options from three dot menu – report editors can choose to expose the chart’s underlying data to the viewer (and the source data) and make it downloadable, or decide to hide it (through the “Hide source data” option in Report sharing settings).
• Omniscope views can now be saved and downloaded as images from the same chart menu (in PNG format).
• Convert and publish Omniscope multi-tab report as PowerPoint presentation via the new custom Report to PowerPoint block (available on Business/Enterprise plans).

Advanced features

Report variable influencing a live SQL query. To use this feature, the database block must be configured for live query and you must tick “Enable live query injection” in the report variable configuration. This will allow you to use the variable name as a parameter in the Database block custom SQL option. See the article 

Omniscope File Drop facility: A new Custom view allows the report viewer to upload a data file by uploading it to a Report page and immediately trigger a data Workflow execution of specific blocks in one request (where the uploaded data file will become an input data block), optionally also refreshing the report which contains the ‘upload file’ view. See more

See also – relevant features already available:

Parameter passing – Custom views in the Report section allow the analyst to integrate a parameter-passing mechanism, so that the report viewer is able to make a choice or enter a new value to be passed to the workflow and trigger a re-execution.
See more

Performance improvements

When multiple users are requesting the same query Omniscope has an option to store and retrieve results of the first-executed query. Report queries are cached indefinitely, subject to a configurable cache memory capacity via an option in the Admin section, and do not otherwise expire until the server app is shut down. Manual report “warm-up” (after data refresh) can be useful on servers with complex reports and large numbers of viewers to facilitate dramatically improved load time and interaction time for common initial actions.

Important Note about the Omniscope Classic Mobile mode sunset – for the users who wish to share their reports as web links Evo is a superior web app – it is better positioned to deliver web based reports and share content. As of this Rock build we will remove support for the old Classic ‘mobile mode’. Please get in touch if you wish to discuss the migration options and licensing.

Let us know what you think, here is a Survey link

#TeamVisokio

The post Omniscope Rock Build – 2023.1 appeared first on Visokio.

App tour

Firstly – we wanted to revisit and improve the user experience – whether the user is an experienced analyst or a total novice. The app is more welcoming and offers help to discover the features, while at the same time facilitates a more intuitive interaction with different interfaces.

We’re rolling out the red carpet – our new users will appreciate an application tour popup – a dismissible sequence of hints to guide them through the basics on the Welcome screen, in the workflow app and on the report pages. Not a paperclip in sight!

Improved workflow building UX

The busy workflow builders will notice the new responsive magnetic auto-connection between the workflow blocks, when adding a new block or moving existing blocks around,

The whole blocks‘ palette can be kept open and pinned to the left, as a collapsible panel, to facilitate faster model building.

Add a new data source to an Omniscope workflow in two seconds: drag and drop a source file directly onto the workflow, or drop it onto a File block options screen to upload it to Omniscope.

Dealing with date patterns has gotten easier with the date format dropdowns in the Field Organiser and Report field settings, where the user can search for a pattern or enter a new one to the list.  

Smooth creation of beautiful reports

Once all your charts are in place – why not give your dataviz a quick makeover?
When optimising the report layout for consumption on different device sizes, you can leverage multiple settings to create the best layout(s) for your report:

• You can use the Free-form layout to position, resize, align and layer charts, filters and objects on the page.
• The Multi page option is there when you need multiple scrollable pages in the same tab. The best choice for producing reports that look like documents.
• Regarding the page sizing behaviour you can choose whether your dashboard will keep fixed size and ratio or to adapt and auto stretch to the screen size.

Free form layout

The perfect view proportions are achieved easily in the ‘Free form’ layout with the new snap views feature.  (Select it from the Layout toolbar menu).

It is a user-friendly way to manipulate the charts on the page and create pixel-perfect dashboards by using chart sizing guide lines, while displaying both pixel count and the chart size as a % of the tab width. 

Colours, fonts, borders, templates

The Report styles section now allows you to pick from a list of pre-configured style presets. You can also create your own presets and share them with colleagues as new Report styles (in XML format). The settings packed in these presets are data default colours, custom fonts, report background colours, elegant borders and spacing, as well as button and logo colours. 

Projects page banner

Even the report list page got a bit of a makeover, with the ability to add banners with text, images, gifs and links using Markdown syntax.
Simply add a banner.md file to the projects folder. You will be able to add instructions or display a welcome message for the report consumers when they find themselves facing a long report list…
Read more

App translations

For our users in Asia we’ve improved “Simplified Chinese” translation, so they can interact with the translated menus and application components. We also support custom translations – read more here and let us know if you need help to implement. More info here.

Test and comment – let us know what you think!

The post Improving Omniscope UI and UX appeared first on Visokio.

No matter the topic or the vertical, once you know where is your data, in any form or shape, Omniscope will provide you with a unified data platform and a simple UI to build and deploy customised web apps for data processing, analytics and reporting. And you don’t need to know how to code.

You may have some questions, as starter “how do I access and load data“? Well, Omniscope has enough core workflow blocks to access files, web resources, databases, services, in various formats, from flat files to structured csv, Excel, JSON and XML. You just need to add and configure a block, once, and re-use it whenever you need to.

At this point you could ask: “I’ve got the data, but how do I clean it, see? It’s all messed up.” Well (2), you’ll find a rich data preparation section to solve the usual problems,  like duplicated records, misspelt words, missing values, and to perform typical operations like pivot, transpose, join, append, aggregate, sort, filter and validate data. You will have enough routines for data prep and ETL, to build workflows to load, stream, blend and transform any data.

“But what about data analytics and machine learning, AI?” In the Analytics section we’ve bundled quite a few outstanding blocks, to deal with the most common data problems solved with classic math, statistics and machine learning algorithms. You will be able (point and click) to perform regression to provide causal analysis or to predict the value of missing data. You’ll find groups (clustering), find associations between them (basket analysis) and anomalies in your data. And more, like leveraging time series forecast, network analysis, text analysis and topic modelling. I am sure you will have enough no-code-data-weapons to extract insights.

“Sure, but how do I build the app UX and UI to share my findings?”
Dataviz? All drag-and-drop, from a set of predefined charts and views, you will be able to build reports, dashboards, or simply interactive web content / pages / tabs, allowing your end user to navigate and interact with graphs, select and filter data, drilling down, and even editing and inputting data if you wanted. You will build the final UX for your end users and embed the customised UI on your site or, viceversa, embed any other web component into the Omniscope “report” single page web app.

“Cool, but does it scale?” Data volume should not be a problem. If you need to process big data, data preparation will leverage all your machine / server resources, in a batch streaming fashion where possible. And if your data is on a SQL database, no problem, you will be able to directly live query data, and leverage your cloud SQL DB to power your dashboards.

“OK… what about automation?” It’s the key, right? Once you build your workflows, web reports and dashboards you want to make sure everything comes to life, showing the recent version of data, and reacting on events. The Scheduler app and the Automation API will be there to run parameterised data refresh & report updates, to trigger tasks, data alerts, and configure, edit & query data. You’ll have all the tools to have Omniscope talk to other software or services, and become your friendly robot. You’ll instruct it once, and it’ll do the job for you. while you can use your time wisely to invent more.

“Sounds interesting, so…. where do I run it, do I need a particular machine or device?”
Omniscope is a web app. It was designed to be universal and deliver a fresh and smart experience on any device: Windows, Mac, Linux, Android, iOS, you just need a web browser, simple (…though better if recent Chrome, Firefox and Edge).
And regarding deployment and security, don’t worry, we won’t see your data. You can install Omniscope on-premises or on your private cloud (AWS, Google Cloud or Azure), and control data and user access / permissions using modern protocols like OIDC and SSO.

The final touch, before you ask: it’s a white-label software: you can host branded data solutions and embedded analytics, get rid of that Visokio Omniscope logo and put your supa-dupa company branding in place. We don’t care about our branding, we only care about building and giving you the best tool for your business.

Ah, and of course, if you are a developer or you’ve got some magic code, be my guest! We made sure you can extend Omniscope and build your own data blocks and custom UI, to augment the data pipelines with your Python / R data transformation and ML/AI scripts , and enhance the web UI with any HTML+JS web components and/or charts.
So you can really achieve the “omni scope” and deliver custom branded data solutions bundling your secret sauce.

Does all this sound like what you’re looking for? Drop us a line, check some demos or give it a go on our public sandbox.

Peace out!

The post No-code Development Platform to build data web apps appeared first on Visokio.

 Not necessarily… Sometimes the winning formula will combine good old tools with innovative solutions. Omniscope provides an environment where any organisation can continue to use Excel for the tasks it’s good for and make it work as a  productive component inside a modern data ecosystem. Xlsx and csv spreadsheets are useful data containers, which are easy to access and edit, but this convenience will inevitably take a toll on the company’s resources, if there is no automated ETL process to take the data from the source to a destination without the human intervention.

How to import, transform or blend Excel data in Omniscope?

a) Super speed: Importing Excel or csv spreadsheets in 2 seconds – just drag and drop a file onto the projects’ list page, or directly onto the data workflow canvas. Omniscope will load the file in the same folder and automatically visualise it! From there you can combine these inputs with data in other formats, from various sources (databases, live feeds or APIs…)

b) Live connection: connect to an Excel or csv spreadsheet on a local or network drive and preserve the connection. Browse to a file location from the File input block. You will be able to establish a live link, so any edits to the file in that location will be reflected and the new data will follow the same transformation process in Omniscope. This can happen on demand – by clicking on the ‘refresh button’ – or via an automated process, using the Scheduler application.

c) Octopus mode: connect to a folder and extract multiple files in one step! If the data storage mechanism collects multiple Excel/csv files in one folder there is no need to drag 52 weekly file blocks to the workflow to get the full picture of your annual data. The ‘Batch append folder’ block can facilitate the extraction of any number of Excel, csv or text files, adding the source field to show where each record came from. (Very useful for audit purposes!) By using the wild character the user can target specific file names or extensions.

 d) Any file, any location – one block to append them all: Extract multiple files from different folders. The ‘Append Files’ block facilitates the extraction of multiple files from any location (machine, network or cloud location) and in any format (not just Excel – CSV, JSON, XML, etc). By default Omniscope will auto-detect the file type and read the contents of the file according to the default options for that file type. The user can configure how Omniscope will read the files by un-ticking ‘Auto detect file type’, then selecting the file type and further options.

Transformation phase – ETL and data quality evaluation

How can the analyst make sure that all those imported Excel files don’t contain any ‘fat finger’ entries, or a messy copy-paste result? Anyone who’s spent more than one Monday morning looking for a typo in a big table will understand that more important than getting the answers fast is making sure that those answers are correct.

You can combine multiple data quality tests inside an Omniscope workflow – start by adding a ‘Validate data’ block to your ETL process. It will screen the data schema (field list and data types), and also evaluate the record count, or the cell values in the target fields. If anything is wrong – a detailed data anomaly list will appear in the block’s diagnostics tab. You will not even have to worry how to remove duplicates – the operation is just a block away… And will be applied every time the new data is appended. More on data quality methods – read here.

Publishing the results

What happens after you’ve imported, validated, augmented and transformed spreadsheet data in Omniscope? You can customise and produce multiple outputs – share clean data with your team, as well as create interactive graphs to allow the users to explore the findings.

There are also options for mass publishing – for those serving hundreds of customised outputs. Use the ‘Batch Output’ block’s spreadsheet interface to define the filtering variation for the data outputs, or ‘Report Generator’ block to create dashboard variations e.g. showing just one country per report.

What about the price of this additional software? Excel is cheap… Think again – Excel is not automating your data management! The amount of man (and woman) hours lost on the manual data transformation / cleaning (copy/paste, rename, search/replace, pivot, then again?), and the time taken fishing for the data faults adds up to more cost than most modern data management tools. By far the biggest danger to an organisation is the cost of not having the right information at the right time or having incorrect data as a basis for decision-making!

Humble spreadsheets are here to stay, however, you may wish to make changes to how Excel is used and integrate it inside a modern data management system to allow it to excel!

The post Excel formula for success is +1: learn how to complement it with Omniscope appeared first on Visokio.

Bring in the Robots!

Save time with the Macro functionality – create reusable workflow logic, then add it as an ETL routine to other projects. 

The new Rock build adds the ability to decompose complex workflows into smaller “macros” and optionally share them with other users/ folders/ projects on the server via the Macros folder.

You can use macros without sharing them, as a way of reducing the complexity of a large workflow, by creating a smaller project alongside the main project, and using the “Macro” block to embed it within the main project.

Alternatively, you can create projects within the dedicated “Macros” folder, and they will become immediately available to all users and projects in other folders on the server, via the Add Block menu.

Read more:

• Macros in Omniscope Evo
• How to create an Omniscope macro

Optimised report mobile experience

Design dazzling dashboards for different devices – preview multiple layouts and optimise every user’s experience.

The report creator can share a link to a dashboard, while making sure that the mobile display will be optimised and look great on any device.

The slider preview device is a switch between different screen sizes, e.g. desktop, tablet or phone, allowing the designer to see what the report will look like on someone’s screen. Their report page is shown within bounds of the chosen target preview device.

This design optimisation process will manage device layouts per tab, so that each tab can have different content, i.e. different set of views. Instances of views added in one of those layouts can also be added in another layout for the different screen size from “Add view” dropdown. A report shown on a big screen may have 10 charts, while a small laptop may show 6, and when the same report is opened on a mobile phone – it may display only 4 charts.

See how to build a report for any device with different layouts

Custom Blocks Library

There is now a better way to manage your own custom blocks – preview, update and delete blocks from a central repository called the Blocks Library. The library is maintained from a single folder on disk and every Custom Block (a subfolder comprising Python or R code and JSON metadata) that is added to this folder is available immediately from the block picker. When library blocks are dragged from the block picker to the workflow, they are linked with the Custom Block source file on disk. Any change to the source file is immediately propagated to all linked blocks in all workflows, thus removing the necessity to update them manually.  You can optionally use VCS such as a GitHub working copy for the Blocks Library folder.

Expanding data connectivity and cloud technologies compatibility

If the sky’s the limit and you’re looking to move your reporting to the cloud – you will appreciate the newly-added support for cloud file systems and file types.

We’ve added support for reading and writing to Google Cloud Storage and Amazon S3 buckets, via an option in the ‘File location’ drop-down (in File and File Output blocks), also in the Batch Append and the Append Files blocks.

Another addition is support for reading and writing Apache Avro and Parquet files.

(read here ​​Apache Parquet read/write support)

Added Batch HTTP block: Connect to any data source with the REST API via the Batch HTTP block, added to handle data-driven HTTP requests. Allows you to precisely configure and execute HTTP requests, where everything can be data-driven: custom headers, request/response payload, URL structure, HTTP method.

Bundled Admin dashboards

You will be able to enjoy Omniscope logs data from the comfort of your own admin dashboard – to understand which reports are getting the most hits, pinpoint which scheduled tasks need attention, or monitor the app performance.

Easy data access with instant data visualisation/exploration

Adding ability to drag and drop data files onto the Workflow app and the Projects page. These files are then automatically uploaded, and a file block is created for each uploaded file.

Capture, save and manage fast-flowing data

Help is on the way if you’re dealing with a ‘data firehose’ scenario: the new Data Savepoint block (from the Operations blocks menu) is allowing you to capture and analyse the data whenever you wish. The block enables you to retain execution data indefinitely at any point in your workflow, so you can continue working and building the workflow downstream of a slow and complex upstream workflow.

Savepoint block has Retention option which will set the behaviour to either:

• “Persistent” – which captures data indefinitely for future executions, or
• “Temporary” – which captures data temporarily only, within a single workflow execution, to ensure reuse of the same data downstream if there are multiple pathways.

Report auto-refresh

New file data refresh option enables you to configure your Report to auto-refresh on open (see Report settings > Global tab). The first person (in a configurable time period) to open this report will trigger the workflow to execute and extract the latest data from the source, and update the visualisation. Useful for infrequently opened projects, or those without a Scheduler-enabled license (a Scheduler is needed if you wish to avoid your users having to wait for the data refresh to complete).

Diagnose query

Report creators will appreciate that in a report with multiple queries they will be able to quickly diagnose which filter combinations are used for each of their queries, and the impact of a complex combination of move/keep actions and filters.

(see Report settings and View data source section).

Dynamic ranking

Calculate your TopN on the fly (based on filtered data): New blocks on the Report data sources menu – Sort and Limit will allow you to display a ready-made queried, calculated, aggregated and ranked dataset in any view. If you wish to dynamically rank top 10 performers, while allowing the report viewer to exclude some data, then you will want to try this mechanism in your reports.

In the screenshot below, our original dataset with ‘all shipment transactions’ has multiple operations applied – the Query block will enable filtering (on the original dataset ‘All shipments’) and control the flow of data into the Aggregation block, where the transactions’ values are summed, then sorted, and a limit is applied to determine the cut-off point for the resulting dataset (to show just Top10 shipments). Every view in this report will have access to this newly-created dynamic dataset.

Chart data download

When making a chart selection, a details button shows in the main toolbar. When clicked – a table showing the selected records is displayed, with options to copy/download the data.

There is also an option to preview the data in each view, showing a table with the displayed and source data and options to copy / export the data.

Commercial plan flexibility

• Adding Team plan licensing support for distinct fixed numbers of Viewers vs Editors concurrent seats.
• New option to create a limit of one session per user. If ticked, only one browser session at a time is allowed for each authenticated user. If the licence has limited seats, this will ensure one user cannot consume multiple seats using different devices, and it will allow installations to support named seats in addition to the default shared seats setup.
  

General & admin

New ability to restrict what resources projects can access on the file system. Read more.

• Better disk and memory management, admin UI options, diagnostics and disk cleanup for the data engine
• Better logging and improved Workflow Jobs Queue admin page, to show the reason jobs were started and cancelled, and when they were created;
• Added option in the admin interface to specify additional arguments to Pip, the Python Package Installer, so that proxies, features, certificates, etc. can be defined.
• Updated Log4J version to 2.17.1, for peace of mind. This update addresses various security vulnerability concerns present in older versions (even though in practical terms they were not vulnerabilities in Omniscope due to the restricted manner in which Omniscope uses Log4J)..
• Omniscope now records resource monitor data every 60 seconds to a new log file “resource_monitor.csv”. Previously available only for the last 7 days via an Admin page, it is now persisted indefinitely as a rolling logfile. Includes information such as system CPU load, memory use, swap, process heap and non-heap, open file handles, and free disk space. Useful when correlating system load with Omniscope activity.
• More bundled demos – find them on the Welcome page; on upgraded existing installations, install new/updated demos from Admin. Alternatively – visit our refreshed  demo gallery

Workflow

• Showing ‘Last Executed’ time when hovering on blocks and on the Report ‘Refresh data’ button to show when the report data was last updated.
• Bookmarks: Ability to create private block bookmarks (available from create bookmark dialog). Private bookmarks will only be available to the user who has created them.
• Added option to download an ioz file without the report data
• Improving performance of large projects with complex workflows
• Updated Google Campaign manager to v3.5.
• Data Table block: option for in-memory (non-persistent) storage
• Field Filter: A new “Auto-populate” button appears when there are no filter rules configured. It inserts a rule for every existing field in the input, letting you easily customise the rules and protect against future field changes or additions.
• File block (and other blocks that let you browse for a file in Omniscope): Added additional sortable columns “Modified”, “Size” and “Type” to the File Browser.
• File and File Output blocks: read/write remote files over SSH (SFTP, SCP) using an optional private key
• File and Append Files blocks: Added options to retry block execution on I/O error
• Field Organiser: Improving performance, including optimising date format conversions. Easy date formats, via a drop-down menu of common formats, while allowing you to continue entering custom formats.
• Promoting workflow execution stats recording to a non-labs feature. A few rock builds ago, we introduced a labs feature that logs workflow-wide and per-block execution time statistics to a CSV logfile, allowing you to analyse bottlenecks in your workflows. It’s now proven for production use with negligible overhead, and has graduated to non-labs.

Reports

• Being able to hide report tabs in the Present mode.
• Supporting 3rd party IFRAME contexts for SSO and Unlimited Viewers licenses, when accessed via HTTPS. However, note that 3rd party IFRAME contexts with single-sign-on (OpenID Connect) are “on notice”, with modern browsers phasing this out over the next 2 years; you should instead ensure your IFRAME embeds are seen as a “same site” as a long-term architecture.
• New options for the spacing/ padding/ border options and behaviour related to creating dividers around the charts

Visualisations

Adding option to show the colour key at the top or bottom of the view, with configurable number of lines for the colour key text.

Bar view:

• Improving multi-tier split labels (>1 split)
• Adding option to set Bar split min/max values if numeric or date continuous axis.
• Adding option to always show stack labels, even if the stack is very small.
• Improving performance of Bar queries when stacking and colouring by stack.

Scatter view:

• Added option to create reference lines in both X and Y axes.
• Added new ‘Jitter’ option – moves markers by a small random amount to provide a better visual sense of the number of observations when either the X or the Y axis are discrete.

Table view

• Added option to limit editing to pre-configured values. A dropdown, rather than a text field, is used to edit the values. Read more here.
• New option to disable editing of number/date fields
• New option to style cells and headers differently for each column

Map view: Added UK NUTS levels 1-3 GeoJSON base layer maps

Sankey view: new option to show percentages

Timeline and Item views are now non-experimental 

New experimental view: Bubble view

and that’s all folks!  Your feedback, as always, is very appreciated, and we are here to support you.

#TeamVisokio

The post Omniscope Rock Build – 2022.1 appeared first on Visokio.

What separates crypto from traditional stocks and derivatives is that all transactions, data and information is immutable on the block chain, to be gathered and seen by any. By using a simple data tool, Omniscope, and the raw data gathered directly from the bitcoin block chain we can produce graphs and set conditions for relatively accurate buys and sells. This article explores on-chain metrics, market sentiment, block chain address activity and several supply and profit metrics. 

Supply in Profit

To start with we take a look at the supply in profit metric. This indicator is shown as a percentage with a lower band at >50% and upper band at <95%. In general when taking a macro view of the market, it is important that not too many investors are sitting on profit if you want the asset to climb in price. As more market participants become profitable, there is a higher likelihood of them selling. When the supply in profit reaches 95% or above, the market is due for a correction. As seen in the view below, the last times this occurred was during the latest price peak in November 2021 and the top of the previous leg back in April 2021. December 2018 was the last time that the % supply in profit dipped below 50%, and was indeed the best buying opportunity in the last 5 years.

% of inactive supply

The chart below shows the % of total supply inactive for 1 year or more. More bitcoins remaining inactive means that they are being held instead of traded, and indeed leads to higher prices being found shortly after. More specifically, if we were to take a look at each time the % of supply inactive reaches new all time highs after finding a macro bottom, we would see that it represents a shift in sentiment and price action. Historically each time a new high is made with this metric, it acts as a leading indicator for future bullish price action. First seen half way through 2012, a new high of 35% of all bitcoins remaining inactive for over a year signaled the macro bottom of that bearish period, and those prices were never seen again. Similarly in January of 2015, bitcoin supply inactive reached a new high of 55%, only a month shy of the macro bottom for that specific bearish period. The last time a high was formed was in July of 2020, preceding the recent run up to $64k. This metric is an excellent indicator for long term macro trend shifts, and should be on  every trader’s radar as a gauge of upcoming momentum shifts and large price moves. Currently this metric reads 62.5%, a level to keep an eye on as we head close to previous highs. 

Net Unrealized Profit/Loss

Next we take a look at NUPL, one of the more popular macro indicators for determining market sentiment. It calculates the total amount of profit or loss in the market. The NUPL value oscillates between 1 and -1.5, and contains bands within each representing market sentiment at that point. The market is in a state of euphoria when the NUPL value is above 0.75 and is shown by the colour blue. Green stands for belief, orange hope and red capitulation. Below is a view of the NUPL plotted with price after bitcoins second halving. Note the capitulation values shown by the NUPL metric are at the exact bottoms of price during the 5 year period. Furthermore, the cycle top of 2018 could have been predicted by the <0.75 reading on the NUPL, turning the bar blue. Selling at euphoria and buying at capitulation seems to be a solid macro strategy. 

As seen from the view, euphoria states are almost never reached, with the last time being in December of 2017, at the peak of the previous bull market. Previously euphoria was reached in 2013 and 2014, both occurring at the absolute top price points of their periods. The last 2 capitulation periods found in the data were the march 2020 crash and the 2018 bear market trough. These were the best buying opportunities for bitcoin in the last 5 years. 

Fear and Greed

Finally, we explore the fear and greed index. The cryptocurrency market is very emotional, consisting largely of retail investors easily spooked by negative price action. Since the asset class could still be considered in its infancy, the lack of liquidity on the order books leads to massive sell-offs in times of crisis or extreme fear. The retail reaction to fear is to sell and “take what you can while you still can”. Similarly a rising market brings strong emotions of greed and FOMO, oftentimes leading investors to overexpose themselves to risk through leverage and derivatives. During these times, the market is usually due for a correction. 

Since the market fluctuates between fear and greed, never staying at one end of the spectrum for too long, we can map the data and extrapolate the best course of action based on the values of this index. Data is gathered from five sources (volatility, market momentum, social media, dominance, trends) and combined to give a fear and greed value between 0 and 100. 

On the view above, the fear and greed values are shown by their marker colour, red indicating a fearful value and green a greedy value. It is clear to see when mapped onto price, that periods of fear are often good buying opportunities. This view shows price action between 2020 until current day. It seems that a cluster of 2 or 3 fearful bars are often followed by a mark up phase in price. Select fearful markers followed by the next greedy ones to see the % change in price had you bought at fear and sold in greed.

Here is the interactive report: Crypto Report

… and the video explainer

Enjoy!

N.B. This article does not constitute financial advice

The post Investing in Crypto – insights for a beginner trader with Omniscope appeared first on Visokio.

The first step in the process is thinking about meaningful questions, so a report can be developed to deliver relevant answers.

With the world’s attention span getting shorter, the analyst will also want to deliver these with a minimal number of charts. This is where the data artisan skills become useful…

Open data sources are everywhere and, in many cases, they come in a format that is not accessible to all potential users (xml, JSON, not just Excel), which means that only patient/ data literate users can interrogate the information and extract meaningful insights.

Curious? This tutorial will show you how to build the interactive dashboard presented below. Read more…

Feel free to play with the charts and filters to see the effects!

The post How to build a live data transaction report appeared first on Visokio.

But how about security? How to run a script in a isolated sandbox so you can be sure it’s not going to harm your machine, service or environment.

A while back we’ve introduced the concept of Custom Blocks in Omniscope. Basically anyone can build their own block by using Python or R to expand the set of blocks we bundle and to enable our users to augment their workflow with any scripts, from new analytics algorithms to simple data transformation routines.
However, such scripts have unrestricted access to the filesystem… so you would have to trust your workflow-permissioned users and the designers who create those blocks.
We’ve then decided to make our Omniscope users life easier, and safer!
Omniscope 2021.2 lets you execute all your custom blocks in a Docker container, via a server-wide policy, where the script code is completely isolated from the server – and its file system other than where configured via options (to mount and map single files or folders).

Set up Docker Desktop (Windows/Mac) or Docker daemon (Linux) and you are ready to go. Read more here

Want to try out? Feel free to use our public sandbox on http://daily.omniscope.me . Happy and safe coding!

The post Secure Python / R scripts execution in Docker with Omniscope appeared first on Visokio.

The post Omniscope Evo: 15 seconds explainer video appeared first on Visokio.

New look welcome page and project templates

The Omniscope welcome page has changed! As well as finding your projects and data files, you can now create projects for common tasks using templates, such as for importing from databases or entering/editing data. Recent files are shown separately, so you can quickly find the project you were working on yesterday. The interface is now more consistent and familiar across the welcome page, workflow and report experiences.

Grouping blocks

Tired of zooming and scrolling through large, complicated workflows? Lost in a sea of blocks and connections? Now you can organise sections of your workflow into groups, optionally name them, and expand them with a click when you need to look or work inside.

Animated transitions in reports

Your reports will come to life when your users filter and select within them, now that Omniscope animates most transitions.
As well as being more funky and engaging, it helps perceive the impact of actions upon the data.

New views

Need to see the cumulative effect of a bar chart? The new Waterfall chart lets you do this, with an optional total bar, and with negative (down) as well as positive (up) sections. Also known as cascading bars, flying bricks or bridge.

Still experimental in this rock, and due to graduate early in version 2021.3, are the new Timeline and Item views. Use the Timeline view to show a small number of events on a single time axis, coloured and labelled. Use the Item view to represent a small number of “items” (e.g. seats in parliament) as coloured dots in one of 4 common arrangements.

Workflow performance 

Performance interacting with and executing workflows has been greatly improved. Memory use under high load is better managed. Experimental “labs” options have been added to let you try other performance improvements in development (please send feedback). Should things go wrong, improved diagnostics will help us understand and resolve the problem faster.

Upgraded data engine

The bundled data engine, used to execute queries underlying the visualisations in your reports, has been upgraded, bringing performance, reliability and disk management improvements. Existing servers will need to rebuild their queryable data from execution data (automatically on first access).
You can find documentation and instructions here.

Secure, isolated Custom Block execution using Docker

Custom Blocks let you write your own code to transform and process data, in R or Python, and then share them as bookmarks or community blocks on GitHub. However, such scripts have unrestricted access to the server filesystem, previously requiring you to trust your workflow-permissioned users. Omniscope 2021.2 now lets you execute all custom blocks using Docker, via a server-wide policy, where the script code is completely isolated from the server file system other than where configured via options. Requires Docker Desktop (Windows/Mac) or Docker daemon (Linux) to be installed and running. (read more)

…and more:

Data preparation

• Templated parameters – new powerful type of param that helps with URL / file path customisation (read more)
• The Google Sheets connector lets you pull data natively from one of your cloud spreadsheets
• Native support for more database vendors. Although Omniscope supports any JDBC based connector, there are now additional guided choices for SQLite, MariaDB, Azure SQL Server, Amazon Redshift, Snowflake, Vertica and Actian Avalanche.
• The Database Output block lets you customise the schema on a per-field basis when creating tables during execution.
• In Field Organiser, the new NETWORKDAYS formula function lets you easily calculate working days between two dates.
• If you’re a DoubleClick user, please note, it has been renamed to Campaign Manager.
• Community blocks:
  • This month’s hot giveaway – new Community block: Field Renamer – just feed it a mapping file containing Old Vs New field names (and your dataset) and it will rename the fields. Faster than typing 100s of new names in the Field Organiser block!

• • Slack API web client: it allows you to call all the public Slack endpoints and methods.
  • Azure Data Lake Storage Gen2 Blob Connector

Interface 

• You can now add notes to every block in the workflow. Useful if you want to explain what you were doing in that 5th Field Organiser block!
• Report tabs can now actually be shown as … tabs! We’ve kept the default (a drop-down) for space efficiency, but you can now enable actual tabs (as a separate toolbar, or configured to appear on the main toolbar if space permits).
• In report, the new Styles menu lets you quickly and easily make common style changes to your views, such as setting the font size.
• Sharing links are now accompanied by QR codes. Useful on a large screen or at a show, and when testing on mobile devices.
• IE, Legacy Edge and Opera mini are no longer supported.

DataViz

• Filter devices have better defaults. Choice devices will be the default, where the data is categorical.
• The Map view improvements: you can show arrows on your “Line” layers – useful to show transport flows, for example. A new Layer picker overlay lets your report users choose what layers they see. And you can preview what’s inside GeoJSON files more easily – useful when configuring how the additional data is rendered and merged.
• Bar, Line and other views now let you make drag-rectangle selections to rapidly and easily select multiple bars or lines.
• Many views now let you define custom tooltips in a consistent and powerful way, so you can manage what information is displayed on mouse-over in visualisations.
• Custom Views: “Set parameters and execute” view – it creates an interface where the report viewer can set the new parameter value and trigger a workflow execution

Admin, branding, permissions

• Manual activation in Evo – relevant in cases where ‘online’ activation/deactivation is blocked by the company’s firewall.
• Improved Admin / Scheduler with multiple working on same tasks/settings.
• New scheduler actions: Refresh all Report blocks and Publish all output blocks.
• Fix for Mac activation lost on system OSX software update.
• Fully brandable welcome/project list.
• Permission presets: Simplification of applying permissions. Ability to define globally (Admin section) permission presets which you can then choose from in permissions dialog.

As always, your feedback is highly appreciated, and we are here for you – if you need help, advice or training to put you in a better shape to deal with data challenges.

Let us be your Q!

Love from #TeamVisokio

The post Omniscope Rock build – 2021.2 appeared first on Visokio.