v3.0.1

Security

• Path traversal protection for archive extraction (tar-slip/zip-slip) — Added new centralized msticpy.common.archive_utils module that validates archive member paths before extraction, preventing malicious archives from writing files outside the intended directory. Applied to GeoIP database extraction (tar), Mordor dataset extraction (zip), and Sentinel query pack downloads (zip). (#889)

Bug Fixes

• NumPy 2.4 compatibility — Regenerated 23 pickle test/data files to eliminate VisibleDeprecationWarning from NumPy 2.4+ caused by legacy align=0 dtype parameters. (#890)
• IPython version constraint by Python version — Split IPython requirement to >=8.39.0 for Python 3.10 and >=9.13.0 for Python 3.11+, since IPython 9.x dropped Python 3.10 support. (#890)

Dependency Updates

Runtime

Development / CI

Internal

• Applied Ruff 0.15.12 formatting to Sentinel provider modules, Azure Monitor driver, Cybereason driver, and outliers module (whitespace/style only).

MSTICPy v3.0.0 Release Notes

MSTICPy v3.0.0 is a major release that brings Python 3.13 support, drops legacy
Python versions, replaces deprecated Azure SDK dependencies, adds a new
OpenObserve data provider, and includes significant improvements to the
Cybereason and Sentinel drivers.

⚡ Breaking Changes

Python Version Support

• Dropped: Python 3.8 and 3.9 are no longer supported.
• Added: Python 3.13 is now fully supported.
• Supported versions: Python 3.10, 3.11, 3.12, and 3.13.

Removed: msrestazure Dependency

The deprecated msrestazure package has been removed and replaced with
azure-mgmt-core>=1.6.0. All Azure authentication and management code
now uses azure-mgmt-core and azure.core.exceptions.

If you have code that catches msrestazure.azure_exceptions.CloudError,
update it to use azure.core.exceptions.HttpResponseError instead:

# Before (v2.x)
from msrestazure.azure_exceptions import CloudError
try:
    result = client.some_operation()
except CloudError as err:
    ...

# After (v3.0)
from azure.core.exceptions import HttpResponseError
try:
    result = client.some_operation()
except HttpResponseError as err:
    ...

Removed: kql Extra

The kql install extra (pip install msticpy[kql]) has been removed.
KqlmagicCustom is no longer a supported dependency.

Removed: aiagents Extra

The aiagents install extra (pip install msticpy[aiagents]) has been removed.
The autogen-agentchat integration is no longer included.

Updated: sql2kql Extra

The sql2kql extra now requires mo-sql-parsing>=11,<12 (previously >=8,<9).
If you use sql_to_kql functionality, ensure your environment is updated:

pip install msticpy[sql2kql]

Removed: pkg_resources

All usages of the deprecated pkg_resources module (from setuptools) have been
replaced with modern alternatives (importlib.resources / importlib.metadata).

Updated Azure SDK Compatibility

• azure-mgmt-resource v25 breaking changes have been addressed.
• azure-mgmt-core minimum version raised to >=1.6.0.

Linting Toolchain: Ruff Replaces Black/Flake8/isort

The project now uses Ruff as the sole linter and formatter, replacing
black, flake8, isort, pydocstyle, and pylint. If you contribute to
MSTICPy, update your development setup:

pip install pre-commit
pre-commit install
# Ruff is configured in pyproject.toml

🚀 New Features

New Data Provider: OpenObserve

A new data driver for OpenObserve has been added,
enabling querying of OpenObserve instances directly from MSTICPy.

Installation:

pip install msticpy[openobserve]

Configuration (in msticpyconfig.yaml):

DataProviders:
  OpenObserve:
    Args:
      connection_str: "https://your-openobserve-host:5080"
      user: "your-username"
      password:
        KeyVault:
      verify: true
      timeout: 300

Usage:

import msticpy as mp
mp.init_notebook()

qry_prov = mp.QueryProvider("OpenObserve")
qry_prov.connect(
    connection_str="https://localhost:5080",
    user="admin",
    password="[PLACEHOLDER]",
    verify=True,
)

# Query with time range
df = qry_prov.exec_query(
    'search "error"',
    days=1,
    limit=100,
    timeout=120,
)

The driver supports time-range parameters (days, start/end),
result limits, timezone configuration, and custom timeouts.

Cybereason Driver Improvements

The Cybereason data driver has received significant enhancements:

• More query parameters exposed — additional control over pagination,
  timeouts, and retry behavior:

  qry_prov = mp.QueryProvider("Cybereason")
  qry_prov.connect(
      instance="cr1",
      tenant_id="...",
      client_id="...",
      client_secret="[PLACEHOLDER]",
  )
  
  df = qry_prov.exec_query(
      "{}",
      page_size=100,
      timeout=120,
      retry_on_error=True,
  )

• HTTP 429 rate-limit handling — automatic retry with backoff when
  Cybereason returns rate-limit responses.

• Improved timeout handling — dedicated _handle_request_timeout method
  for graceful recovery from request timeouts.

• Better error reporting — MsticpyDataQueryError now includes exception
  name and args for clearer diagnostics.

• Improved data flattening — _flatten_simple_values and
  _flatten_element_values have been reworked for more reliable output,
  with extra fields now mapped to the result DataFrame.

• Structural pattern matching — query response parsing now uses Python
  match/case statements for cleaner, more maintainable code.

Sentinel Certificate Authentication

Microsoft Sentinel connections now support certificate-based authentication
as an alternative to client secrets, providing enhanced security for
automated pipelines and service accounts. See Sentinel Provider for details.

ThreatIntelIndicators: New Table Schema Support

The Azure Sentinel BYOTI (Bring Your Own Threat Intelligence) provider now
supports both the old and new ThreatIntelIndicators table schemas. This
enables seamless operation across Sentinel workspaces regardless of which
schema version they use.

• Configurable confidence threshold — the TI confidence threshold is now
  a class constant that can be overridden, rather than a hardcoded value.

Timeseries Anomaly Detection Fixes

• Fixed ts_anomalies_stl to correctly pass a Series (not raw values) to
  the STL decomposition, resolving errors with certain input shapes.
• Fixed handling of the seasonal == 0 edge case in timeseries anomaly
  detection.

KQL Timespan Conversion Utilities

New utility functions for converting KQL timespan strings to Python
timedelta objects. Uses azure-kusto-data's parse_timedelta for
accurate KQL-compatible timespan parsing.

AzureSearchDriver Fixes

Fixed the AzureSearchDriver to properly set the MSSentinelSearch
environment and correctly override the query method.

📦 Dependency Changes

Added

Removed

Updated

Removed Python-Version-Conditional Dependencies

• importlib-resources conditional on Python ≤3.8 has been removed.
• Separate ipython version pins for Python <3.8 / ≥3.8 consolidated to
  a single >=7.23.1 requirement.

🛠️ Developer & Tooling Changes

• Ruff is now the single linter and formatter (replaces black, flake8,
  isort, pydocstyle, pylint). Configuration is in pyproject.toml.
• Pre-commit hooks updated to use Ruff for both linting and formatting.
• CI now tests against Python 3.10, 3.11, 3.12, and 3.13 on Ubuntu.
• ReadTheDocs configuration updated to Python 3.12 / Ubuntu 24.02.
• Structural pattern matching (match/case) is used where appropriate
  (requires Python 3.10+).
• Modern type hint syntax throughout (e.g., list[str] instead of
  List[str], str | None instead of Optional[str]).

🐛 Bug Fixes

• Fixed driver connection error messages in Splunk driver and tests.
• Fixed sporadic WhoIs test failures.
• Fixed Self type import for compatibility across Python versions.
• Fixed check_cli_credentials to use up-to-date Azure authentication code.
• Fixed provider name handling in TI lookup objects.
• Fixed result_format type annotation in resource graph driver.
• Fixed linting issues across the codebase for Ruff compatibility.

📖 Documentation

• Added documentation for Sentinel certificate authentication.
• Updated documentation to use modern type hint syntax.
• Sphinx API docs updated for new and changed modules.
• Added GitHub Copilot instructions file for contributors.

🙏 Contributors

Thanks to all contributors who made this release possible:

• Florian Bracq (@FlorianBracq) — Cybereason driver improvements, query parameter enhancements, and structural pattern matching
• Ian Hellen (@ianhelle) — Python 3.13 support, Azure SDK modernization, tooling migration, and core maintenance
• Julien (@juju4) — OpenObserve data provider

Full Changelog: v2.17.2...v3.0.0

761 files changed, 45,270 insertions(+), 46,385 deletions(-)

What's Changed

• M365D→MDE endpoint consolidation due to Microsoft API deprecation
• Updating Defender documentation to add changes in use of APIs and providers
• Adding use of delegated auth documentation to Defender documentation
• Bug fixes: Azure Monitor datetime handling, OData driver config lookup, M365D queries

Full Changelog: v2.17.1...v2.17.2

This minor release fixes a missing scope property in the standard OAuth login flow for M365D/MDE data providers.

• Fixed OAuth v2 scope handling by adding scope parameter to request body when resource parameter is not present
• Added comprehensive debug and info level logging throughout the authentication and query execution flows
• Extended unit test coverage for odata_driver with tests for connection string parsing, error handling, and various authentication methods

Summary

This release delivers new analytics capability (RRCF outlier detection), expanded cloud detection coverage (Prisma Cloud AWS), and a modernization of authentication by moving Defender data providers to OAuth2 with corrected scopes. It also fixes several reliability issues (MSI auth logic, KQL timezone handling, query value escaping), updates visualization and widget code for Bokeh 3.7 and Python 3.12, tightens typing (mypy/key vault), and refreshes CI/publish infrastructure and supported Python versions. Users of Defender integrations should review and update scopes/config before upgrading.

🚀 New Features

• RRCF Outlier Detection (random cut forest) adds a new anomaly detection method for telemetry analysis. (PR #846)
• Prisma Cloud AWS detection enhancement broadens cloud security coverage. (PR #847)

🔐 Authentication & Authorization Changes (Action Recommended)

• Defender APIs migrated to OAuth2 with corrected/normalized scopes. Update any legacy scope/resource configurations in msticpyconfig.yaml (e.g., adopt .default scopes) before upgrading. (PR #856)

🛠 Enhancements & Refactors

• Bokeh 3.7 compatibility adjustments. (PR #843)
• Query time widget: remove Python 3.12 deprecation warnings + richer typing. (PR #849)
• Key Vault client mypy/type fixes improving static analysis fidelity. (PR #852)

🐛 Bug Fixes

• MSI authentication logic corrected for Managed Identity scenarios. (PR #844)
• Proper escaping of single quotes inside list query elements to avoid malformed queries. (PR #851)
• KQL timezone handling fixed for accurate temporal queries. (PR #850)
• OAuth scope validation/normalization as part of Defender OAuth2 shift. (PR #856)

🧪 Analytics & Data Quality

• RRCF anomaly scoring (PR #846) enables ensemble-based outlier detection.
• Enhanced Prisma Cloud AWS detections (PR #847) improve coverage.

🧰 Developer Experience / Code Quality

• Typing and mypy cleanup (PRs #849, #852).
• Reduced deprecation warnings (PR #849).

🏗 CI / Build / Release Infrastructure

• Azure Pipelines PyPI publish workflow updated. (PR #845)
• Supported Python build versions refreshed. (PR #853)
• PyPI publish action bumped (1.5.1 → 1.13.0). (PR #854)

⚠️ Potential Breaking / Behavior Changes

• Defender provider auth: legacy non-OAuth2 or incorrect scope names will fail until configs updated. (PR #856)
• Timezone fix may adjust timestamp normalization. (PR #850)

📘 Upgrade Checklist

1. Update Defender provider scopes to OAuth2 .default values.
2. Test a minimal Defender query after reconnect.
3. Validate time range widgets for expected timezone behavior.
4. Integrate RRCF if desired.
5. Align local Python with updated CI matrix.
6. Run static checks to adjust for stricter typing.

🙌 Contributors

@ianhelle, @FlorianBracq, @Tatsuya-hasegawa, @raj-axe, @dependabot

PR Reference Index

• Bokeh 3.7 fixes – #843
• Fix logic issue with MSI auth – #844
• Update azure-pipelines-publish-pypi.yml – #845
• Add rrcf outlier – #846
• Prismacloud_aws_detection_enhancement – #847
• [nbwidgets] Querytime Python 3.12 deprecation + typing – #849
• Fix kql timezone support – #850
• Escape single quotes for list elements from queries – #851
• Addressing mypy errors in keyvault_client – #852
• Updating Python build versions – #853
• Bump pypa/gh-action-pypi-publish 1.5.1 → 1.13.0 – #854
• Move Defender APIs to OAuth2, fix invalid scopes – #856

Full Changelog: v2.16.2...v2.17.0

This is mainly a release to fix compatibility problems with some features deprecated in Bokeh 3.7

Also includes fixes for different behaviour of VTObject in VirusTotalV3 code - which messes up conversion to pandas dataframes.

Finally I've added some fixes for using AzureCliCredential and ManagedIdentityCredential. In cases where you
are using AzureCLI authentication with a ManagedIdentity (such as in AzureML compute), the credential
fails if you supply a TenantId when creating. The code now checks that it can obtain a token and, if not, falls
back to creating the credential with no tenantId.
Similarly, the default for ManagedIdentityCredential is now to create it only passing client_id (or None if this is not defined).
It will fallback to previous behavior, if this fails.
If that also fails, it will fall back to creating the credential with no parameters.

What's Changed

• Compat fixes for Bokeh 3.7 by @ianhelle in #840
• Avoiding vulnerable dependencies by @ianhelle in #843
  • updating cryptography to >=43.0.1
  • Add explicit dependencies for jinja2>=3.1.5 and tornado>=6.4.2 to avoid vulnerable versions

Full Changelog: v2.16.1...v2.16.2.post

Highlights

This is largely a "fix-and-improve" release.
Some important fixes to:

• QueryEditor
• Pagination and retry capability added to Prisma Cloud Driver
• Dataclass issue

As of this release we are switching from GitHub actions publishing to an Azure DevOps pipeline
(this is a Microsoft internal security requirement for enhancing supply chain security). It should not
affect your enjoyment of this package :-)

What's Changed

• Fix dataclass issue by @FlorianBracq in #833
• Add typing for Defender by @FlorianBracq in #828
• Edit process_cmd_line function template to accept list as parameter by @vx3r in #835
• Fix to QueryEditor by @ianhelle in #836
• Add pagination and retries based on load, support queries - PrismaCloudDriver by @raj-axe in #834
• Adding Azure Publishing pipeline by @ianhelle in #838

Full Changelog: v2.16.0...v2.16.1

Prisma Cloud Driver

This pull request adds support for integrating Prisma Cloud into MSTICPy. By including a dedicated PrismaCloudDriver, the goal is to enable querying and analyzing data from Prisma Cloud’s APIs within MSTICPy’s data analysis framework.

The Prisma Cloud Driver, developed by Palo Alto Networks, integrates MSTICPy with Prisma Cloud’s security platform. It enables seamless authentication, querying, and data retrieval from Prisma Cloud’s assets, configurations, and events. By incorporating this driver, MSTICPy users gain streamlined access to cloud security data, allowing to perform in-depth threat analysis, compliance checks, and security investigations directly within their existing data analysis workflows

Big thanks to @raj-axe for this

Cyberint TI Provider

TI provider uses the Cyberint API for IoC lookup.

Azure Sentinel/Azure Monitor

We've had a bit of activity around Azure Sentinel/Azure Monitor.
@JPvRiel has been digging into this and found a few bugs. They also raised the issue the current Azure monitor driver
has no support for custom tables. I created an experimental driver in this release but it's not working as expected.
If anyone wants to take up the sword and tackle bugs #829, #830 and #831 I would appreciate your help.
#831 is specifically the problems with the experimental driver
The other two are bugs in the existing Azure Monitor/Sentinel provider. (although I'm not the support for parsing time ranges is an easy fix since we're relying on the azure.monitor.query SDK to do this conversion.

Thanks to @vx3r for this.

Certificate Authentication support for OData drivers (Defender and MSGraph)

Thanks to @FlorianBracq for this.

Other changes

Lots more typing work by our esteemed @FlorianBracq

Various fixes but some important ones:

• Maxmind API change
• Bokeh (should now support current Bokeh versions)
• Panel (workaround for seeming bug in 1.16.1)

What's Changed

• Fix typing issue for FoliumMap by @FlorianBracq in #814
• Add Azure kusto driver typing by @FlorianBracq in #816
• Odata certificate support by @FlorianBracq in #812
• Fix change to maxmind API 2.6.3 by @ianhelle in #823
• Apply typing to the Cybereason driver by @FlorianBracq in #813
• add Cyberint TI provider by @vx3r in #817
• Ianhelle/update to v2.16.0 by @ianhelle in #824
• Ianhelle/az monitor search driver 2025 02 05 by @ianhelle in #825
• Fixed autogen package by @ekzhu in #818
• prisma_cloud driver by @raj-axe in #821
• Updating bokeh code to support 3.4.0+ by @ianhelle in #826
• Cyberint risk key none value by @vx3r in #832

New Contributors

• @ekzhu made their first contribution in #818
• @raj-axe made their first contribution in #821

Full Changelog: v2.15.0...v2.16.0

Highlights

Multi-dimensional plots for outliers by @Tatsuya-hasegawa

The outliers module has lived in MSTICPy for a long time but been some neglected
@Tatsuya-hasegawa (hacker-T) has contributed some cool visualizations to
better interpret the data.
Many thanks!!!

import numpy as np
from msticpy.analysis.outliers import identify_outliers,plot_outlier_results

n_dimension = 7

# create random numeric samples
data = np.random.rand(100, n_dimension)

# calc outliers by Isolation Forest algorism
clf, X_outliers, y_pred_outliers = identify_outliers(data, data, contamination=0.1, max_features=0.4)
    
feature_columns = [f'feature{i}' for i in range(1, n_dimension+1)]

plot_outlier_results(
    clf, 
    data,
    data, 
    X_outliers, 
    feature_columns=feature_columns, 
    plt_title="MSTICPY Isolation Forest Anomaly Detection for Multi Dimension Features"
)

Improved code/docs for federated authentication for M365D/M356 Graph providers - @ryan-detect-dot-dev

Although using federated auth (rather than client secret) has been possible for a while, the documentation
for how to use this was in the MSTICPy docs. Thanks to Ryan we now have this (along with cleaned up code
for the Defender* data providers.
(although Ryan is listed as a new contributor below - he has made several previous contributions under
a different GitHub identity)

Rigorous Type Annotation work started by @FlorianBracq earlier this year continues.

This helps to make the code more robust and clearer to read and use. This is thankless work but my
huge thanks go out to @FlorianBracq for this!

Other fixes

Some other important fixes to CyberReason driver and Azure Monitor/MS Sentinel driver are also included

What's Changed

• Cybereason driver fix http429 tests and exception by @vx3r in #803
• Cybereason driver query return instance name in dataframe by @vx3r in #804
• Add multi dimension plots to analysis.outliers module. by @Tatsuya-hasegawa in #805
• Avoid httpx 0.28.0 for unit tests by @ianhelle in #811
• Add typing hints to core classes by @FlorianBracq in #810
• Fixing azure_monitor_driver for deprecated httpx API by @ianhelle in #809
• Update version to 2.15.0 by @ianhelle in #806
• Update MDATP Driver for delegated auth by @ryan-detect-dot-dev in #784

New Contributors

• @ryan-detect-dot-dev made their first contribution in #784

Full Changelog: v2.14.0...v2.15.0

User Session Configuration

Do you always have one or more data providers or other components that you need to load for every notebook you create?
I do, and got a bit fed up with typing the same lines of code over and over again.

User session configuration lets you specify which providers are loaded, whether or not to connect and which parameters
to supply at load and connect time. You put all of this into a straightforward YAML file and load it using the following:

import msticpy as mp   # you likely will already be doing this
mp.init_notebook()     # and this

mp.load_user_session("my_config.yaml")   # if you have a "mp_user_session.yaml" in the current directory
                                         # you can skip the parameter

This example shows the structure of the YAML:

QueryProviders:
  qry_prov_sent:
    DataEnvironment: MSSentinel
    InitArgs:
      debug: True
    Connect: True
    ConnectArgs:
      workspace: MySoc
      auth_methods: ['cli', 'device_code']
  qry_prov_md:
    DataEnvironment: M365D
Components:
   mssentinel:
      Module: msticpy.context.azure
      Class: MicrosoftSentinel
      InitArgs:
      Connect: True
      ConnectArgs:
          workspace: MySoc
          auth_methods: ['cli', 'device_code']

The providers/components created (e.g. qry_prov_sent in this example)
are published back to your notebook Python namespace, so you'll see
these available as variables ready to use.

This configuration file is equivalent to the following code:

qry_prov_sent = mp.QueryProvider("MSSentinel")
qry_prov_sent.connect(workspace="MySoc", auth_methods=['cli', 'device_code'])
qry_prov_md = mp.QueryProvider("M365D")

from msticpy.context.azure import MicrosoftSentinel
mssentinel = MicrosoftSentinel()
mssentinel.connect(workspace="MySoc", auth_methods=['cli', 'device_code'])

Not a huge saving, on the face of it, but if you create a lot of notebooks or want to use
msticpy in an automation scenario, it can be very helpful.
Include a verbose=True parameter to load_user_session to see more detailed logging of what is going on.
See the full documentation here

Maxmind GeoIPLite fix

Sometime recently (not too sure when) Maxmind changed their download procedure to use
a different URL and authentication mechanism. This was causing auto-update to fail. To use
the new mechanism you need to get your Maxmind User Account ID (login and look at your
account properties) and add that to your msticpyconfig.yaml as shown below.

OtherProviders:
  GeoIPLite:
    Args:
      AccountID: "1234567"
      AuthKey:
        EnvironmentVar: "MAXMIND_AUTH"
      DBFolder: "~/.msticpy"
    Provider: "GeoLiteLookup"

Extract nested dictionaries from pandas column to multiple rows/columns

@pioneerHitesh has added this as a new method in the mp_pivot pandas extension:

data_df.mp_pivot.dict_to_dataframe(col="my_nested_column")

It returns a dataframe with the column recursively expanded:

• lists become new rows
• dictionaries become new columns

So a column with the following structure:

my_df = src_df.mp_pivot.dict_to_dataframe(col="NCol")
my_df

Would be unpacked to:

What's Changed

• Authentication module unit test by @ianhelle in #800
• Use sessions config and GeoIP download failure by @ianhelle in #801
• Added Inbuilt function to extract nested JSON by @pioneerHitesh in #798
• Add max retry parameter to the execution prevent HTTP 429 by @vx3r in #802

New Contributors

• @pioneerHitesh made their first contribution in #798
• @vx3r made their first contribution in #802

Full Changelog: v2.13.1...v2.14.0