Skip to content

8.6.0 #557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

8.6.0 #557

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ __pycache__/
# Distribution / packaging
.Python
env/
.venv
build/
develop-eggs/
dist/
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# DeepDiff Change log

- [Unreleased]
- Colored View: Output pretty-printed JSON with color-coded differences (added in green, removed in red, changed values show old in red and new in green).

- v8-5-0
- Updating deprecated pydantic calls
- Switching to pyproject.toml
Expand Down
79 changes: 79 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

DeepDiff is a Python library for deep comparison, searching, and hashing of Python objects. It provides:
- **DeepDiff**: Deep difference detection between objects
- **DeepSearch**: Search for objects within other objects
- **DeepHash**: Content-based hashing for any object
- **Delta**: Git-like diff objects that can be applied to other objects
- **CLI**: Command-line interface via `deep` command

## Development Commands

### Setup
```bash
# Install with all development dependencies
pip install -e ".[cli,coverage,dev,docs,static,test]"
# OR using uv (recommended)
uv sync --all-extras
```

### Testing
```bash
# Run tests with coverage
pytest --cov=deepdiff --cov-report term-missing

# Run tests including slow ones
pytest --cov=deepdiff --runslow

# Run single test file
pytest tests/test_diff_text.py

# Run tests across multiple Python versions
nox -s pytest
```

### Quality Checks
```bash
# Linting (max line length: 120)
nox -s flake8

# Type checking
nox -s mypy

# Run all quality checks
nox
```

## Architecture

### Core Structure
- **deepdiff/diff.py**: Main DeepDiff implementation (most complex component)
- **deepdiff/deephash.py**: DeepHash functionality
- **deepdiff/base.py**: Shared base classes and functionality
- **deepdiff/model.py**: Core data structures and result objects
- **deepdiff/helper.py**: Utility functions and type definitions
- **deepdiff/delta.py**: Delta objects for applying changes

### Key Patterns
- **Inheritance**: `Base` class provides common functionality with mixins
- **Result Objects**: Different result formats (`ResultDict`, `TreeResult`, `TextResult`)
- **Path Navigation**: Consistent path notation for nested object access
- **Performance**: LRU caching and numpy array optimization

### Testing
- Located in `/tests/` directory
- Organized by functionality (e.g., `test_diff_text.py`, `test_hash.py`)
- Aims for ~100% test coverage
- Uses pytest with comprehensive fixtures

## Development Notes

- **Python Support**: 3.9+ and PyPy3
- **Main Branch**: `master` (PRs typically go to `dev` branch)
- **Build System**: Modern `pyproject.toml` with `flit_core`
- **Dependencies**: Core dependency is `orderly-set>=5.4.1,<6`
- **CLI Tool**: Available as `deep` command after installation with `[cli]` extra
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ include *.txt
include *.sh
include pytest.ini
include *.py
exclude uv.lock
recursive-include docs/ *.rst
recursive-include docs/ *.png
recursive-include tests *.csv
Expand Down
10 changes: 10 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,16 @@ Please take a look at the [CHANGELOG](CHANGELOG.md) file.

:mega: **Please fill out our [fast 5-question survey](https://forms.gle/E6qXexcgjoKnSzjB8)** so that we can learn how & why you use DeepDiff, and what improvements we should make. Thank you! :dancers:

# Local dev

1. Clone the repo
2. Switch to the dev branch
3. Create your own branch
4. Install dependencies

- Method 1: Use [`uv`](https://github.com/astral-sh/uv) to install the dependencies: `uv sync --all-extras`.
- Method 2: Use pip: `pip install -e ".[cli,coverage,dev,docs,static,test]"`

# Contribute

1. Please make your PR against the dev branch
Expand Down
139 changes: 139 additions & 0 deletions deepdiff/colored_view.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
import json
import os
from ast import literal_eval
from importlib.util import find_spec
from typing import Any, Dict

from deepdiff.model import TextResult, TreeResult


if os.name == "nt" and find_spec("colorama"):
import colorama

Check warning on line 11 in deepdiff/colored_view.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/colored_view.py#L11

Added line #L11 was not covered by tests

colorama.init()

Check warning on line 13 in deepdiff/colored_view.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/colored_view.py#L13

Added line #L13 was not covered by tests


# ANSI color codes
RED = '\033[31m'
GREEN = '\033[32m'
RESET = '\033[0m'


class ColoredView:
"""A view that shows JSON with color-coded differences."""

def __init__(self, t2: Any, tree_result: TreeResult, compact: bool = False):
self.t2 = t2
self.tree = tree_result
self.compact = compact
self.diff_paths = self._collect_diff_paths()

def _collect_diff_paths(self) -> Dict[str, str]:
"""Collect all paths that have differences and their types."""
text_result = TextResult(tree_results=self.tree, verbose_level=2)
diff_paths = {}
for diff_type, items in text_result.items():
if not items:
continue
try:
iter(items)
except TypeError:
continue

Check warning on line 41 in deepdiff/colored_view.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/colored_view.py#L40-L41

Added lines #L40 - L41 were not covered by tests
for path, item in items.items():
if diff_type in ("values_changed", "type_changes"):
changed_path = item.get("new_path") or path
diff_paths[changed_path] = ("changed", item["old_value"], item["new_value"])
elif diff_type in ("dictionary_item_added", "iterable_item_added", "set_item_added"):
diff_paths[path] = ("added", None, item)
elif diff_type in ("dictionary_item_removed", "iterable_item_removed", "set_item_removed"):
diff_paths[path] = ("removed", item, None)
return diff_paths

def _format_value(self, value: Any) -> str:
"""Format a value for display."""
if isinstance(value, bool):
return 'true' if value else 'false'
elif isinstance(value, str):
return f'"{value}"'
elif isinstance(value, (dict, list, tuple)):
return json.dumps(value)

Check warning on line 59 in deepdiff/colored_view.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/colored_view.py#L59

Added line #L59 was not covered by tests
else:
return str(value)

def _get_path_removed(self, path: str) -> dict:
"""Get all removed items for a given path."""
removed = {}
for key, value in self.diff_paths.items():
if value[0] == 'removed' and key.startswith(path + "["):
key_suffix = key[len(path):]
if key_suffix.count("[") == 1 and key_suffix.endswith("]"):
removed[literal_eval(key_suffix[1:-1])] = value[1]
return removed

def _has_differences(self, path_prefix: str) -> bool:
"""Check if a path prefix has any differences under it."""
return any(diff_path.startswith(path_prefix + "[") for diff_path in self.diff_paths)

def _colorize_json(self, obj: Any, path: str = 'root', indent: int = 0) -> str:
"""Recursively colorize JSON based on differences, with pretty-printing."""
INDENT = ' '
current_indent = INDENT * indent
next_indent = INDENT * (indent + 1)

if path in self.diff_paths and path not in self._colorize_skip_paths:
diff_type, old, new = self.diff_paths[path]
if diff_type == 'changed':
return f"{RED}{self._format_value(old)}{RESET} -> {GREEN}{self._format_value(new)}{RESET}"
elif diff_type == 'added':
return f"{GREEN}{self._format_value(new)}{RESET}"
elif diff_type == 'removed':
return f"{RED}{self._format_value(old)}{RESET}"

if isinstance(obj, (dict, list)) and self.compact and not self._has_differences(path):
return '{...}' if isinstance(obj, dict) else '[...]'

if isinstance(obj, dict):
if not obj:
return '{}'

Check warning on line 97 in deepdiff/colored_view.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/colored_view.py#L97

Added line #L97 was not covered by tests
items = []
for key, value in obj.items():
new_path = f"{path}['{key}']" if isinstance(key, str) else f"{path}[{key}]"
if new_path in self.diff_paths and self.diff_paths[new_path][0] == 'added':
# Colorize both key and value for added fields
items.append(f'{next_indent}{GREEN}"{key}": {self._colorize_json(value, new_path, indent + 1)}{RESET}')
else:
items.append(f'{next_indent}"{key}": {self._colorize_json(value, new_path, indent + 1)}')
for key, value in self._get_path_removed(path).items():
new_path = f"{path}['{key}']" if isinstance(key, str) else f"{path}[{key}]"
items.append(f'{next_indent}{RED}"{key}": {self._colorize_json(value, new_path, indent + 1)}{RESET}')
return '{\n' + ',\n'.join(items) + f'\n{current_indent}' + '}'

elif isinstance(obj, (list, tuple)):
if not obj:
return '[]'

Check warning on line 113 in deepdiff/colored_view.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/colored_view.py#L113

Added line #L113 was not covered by tests
removed_map = self._get_path_removed(path)
for index in removed_map:
self._colorize_skip_paths.add(f"{path}[{index}]")

items = []
remove_index = 0
for index, value in enumerate(obj):
while remove_index == next(iter(removed_map), None):
items.append(f'{next_indent}{RED}{self._format_value(removed_map.pop(remove_index))}{RESET}')
remove_index += 1
items.append(f'{next_indent}{self._colorize_json(value, f"{path}[{index}]", indent + 1)}')
remove_index += 1
for value in removed_map.values():
items.append(f'{next_indent}{RED}{self._format_value(value)}{RESET}')
return '[\n' + ',\n'.join(items) + f'\n{current_indent}' + ']'
else:
return self._format_value(obj)

def __str__(self) -> str:
"""Return the colorized, pretty-printed JSON string."""
self._colorize_skip_paths = set()
return self._colorize_json(self.t2)

def __iter__(self):
"""Make the view iterable by yielding the tree results."""
yield from self.tree.items()

Check warning on line 139 in deepdiff/colored_view.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/colored_view.py#L139

Added line #L139 was not covered by tests
8 changes: 7 additions & 1 deletion deepdiff/commands.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
@click.option('--significant-digits', required=False, default=None, type=int, show_default=True)
@click.option('--truncate-datetime', required=False, type=click.Choice(['second', 'minute', 'hour', 'day'], case_sensitive=True), show_default=True, default=None)
@click.option('--verbose-level', required=False, default=1, type=click.IntRange(0, 2), show_default=True)
@click.option('--view', required=False, type=click.Choice(['tree', 'colored', 'colored_compact'], case_sensitive=True), show_default=True, default='tree')
@click.option('--debug', is_flag=True, show_default=False)
def diff(
*args, **kwargs
Expand All @@ -74,6 +75,8 @@
t2_path = kwargs.pop("t2")
t1_extension = t1_path.split('.')[-1]
t2_extension = t2_path.split('.')[-1]
if "view" in kwargs and kwargs["view"] is None:
kwargs.pop("view")

Check warning on line 79 in deepdiff/commands.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/commands.py#L79

Added line #L79 was not covered by tests

for name, t_path, t_extension in [('t1', t1_path, t1_extension), ('t2', t2_path, t2_extension)]:
try:
Expand Down Expand Up @@ -112,7 +115,10 @@
sys.stdout.buffer.write(delta.dumps())
else:
try:
print(diff.to_json(indent=2))
if kwargs["view"] in {'colored', 'colored_compact'}:
print(diff)

Check warning on line 119 in deepdiff/commands.py

View check run for this annotation

Codecov / codecov/patch

deepdiff/commands.py#L119

Added line #L119 was not covered by tests
else:
print(diff.to_json(indent=2))
except Exception:
pprint(diff, indent=2)

Expand Down
27 changes: 21 additions & 6 deletions deepdiff/delta.py
Original file line number Diff line number Diff line change
Expand Up @@ -330,11 +330,21 @@ def _set_new_value(self, parent, parent_to_obj_elem, parent_to_obj_action,
Set the element value on an object and if necessary convert the object to the proper mutable type
"""
if isinstance(obj, tuple):
# convert this object back to a tuple later
obj = self._coerce_obj(
parent, obj, path, parent_to_obj_elem,
parent_to_obj_action, elements,
to_type=list, from_type=tuple)
# Check if it's a NamedTuple and use _replace() to generate a new copy with the change
if hasattr(obj, '_fields') and hasattr(obj, '_replace'):
if action == GETATTR:
obj = obj._replace(**{elem: new_value})
if parent:
self._simple_set_elem_value(obj=parent, path_for_err_reporting=path,
elem=parent_to_obj_elem, value=obj,
action=parent_to_obj_action)
return
else:
# Regular tuple - convert this object back to a tuple later
obj = self._coerce_obj(
parent, obj, path, parent_to_obj_elem,
parent_to_obj_action, elements,
to_type=list, from_type=tuple)
if elem != 0 and self.force and isinstance(obj, list) and len(obj) == 0:
# it must have been a dictionary
obj = {}
Expand Down Expand Up @@ -709,7 +719,12 @@ def _do_set_or_frozenset_item(self, items, func):
obj = self._get_elem_and_compare_to_old_value(
parent, path_for_err_reporting=path, expected_old_value=None, elem=elem, action=action, forced_old_value=set())
new_value = getattr(obj, func)(value)
self._simple_set_elem_value(parent, path_for_err_reporting=path, elem=elem, value=new_value, action=action)
if hasattr(parent, '_fields') and hasattr(parent, '_replace'):
# Handle parent NamedTuple by creating a new instance with _replace(). Will not work with nested objects.
new_parent = parent._replace(**{elem: new_value})
self.root = new_parent
else:
self._simple_set_elem_value(parent, path_for_err_reporting=path, elem=elem, value=new_value, action=action)

def _do_ignore_order_get_old(self, obj, remove_indexes_per_path, fixed_indexes_values, path_for_err_reporting):
"""
Expand Down
21 changes: 18 additions & 3 deletions deepdiff/diff.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@
type_is_subclass_of_type_group, type_in_type_group, get_doc,
number_to_string, datetime_normalize, KEY_TO_VAL_STR, booleans,
np_ndarray, np_floating, get_numpy_ndarray_rows, RepeatedTimer,
TEXT_VIEW, TREE_VIEW, DELTA_VIEW, detailed__dict__, add_root_to_paths,
TEXT_VIEW, TREE_VIEW, DELTA_VIEW, COLORED_VIEW, COLORED_COMPACT_VIEW,
detailed__dict__, add_root_to_paths,
np, get_truncate_datetime, dict_, CannotCompare, ENUM_INCLUDE_KEYS,
PydanticBaseModel, Opcode, SetOrdered, ipranges)
from deepdiff.serialization import SerializationMixin
Expand All @@ -40,6 +41,7 @@
from deepdiff.deephash import DeepHash, combine_hashes_lists
from deepdiff.base import Base
from deepdiff.lfucache import LFUCache, DummyLFU
from deepdiff.colored_view import ColoredView

if TYPE_CHECKING:
from pytz.tzinfo import BaseTzInfo
Expand Down Expand Up @@ -80,7 +82,7 @@ def _report_progress(_stats, progress_logger, duration):
PREVIOUS_DIFF_COUNT = 'PREVIOUS DIFF COUNT'
PREVIOUS_DISTANCE_CACHE_HIT_COUNT = 'PREVIOUS DISTANCE CACHE HIT COUNT'
CANT_FIND_NUMPY_MSG = 'Unable to import numpy. This must be a bug in DeepDiff since a numpy array is detected.'
INVALID_VIEW_MSG = 'The only valid values for the view parameter are text and tree. But {} was passed.'
INVALID_VIEW_MSG = "view parameter must be one of 'text', 'tree', 'delta', 'colored' or 'colored_compact'. But {} was passed."
CUTOFF_RANGE_ERROR_MSG = 'cutoff_distance_for_pairs needs to be a positive float max 1.'
VERBOSE_LEVEL_RANGE_MSG = 'verbose_level should be 0, 1, or 2.'
PURGE_LEVEL_RANGE_MSG = 'cache_purge_level should be 0, 1, or 2.'
Expand Down Expand Up @@ -365,7 +367,11 @@ def _group_by_sort_key(x):

self.tree.remove_empty_keys()
view_results = self._get_view_results(self.view)
self.update(view_results)
if isinstance(view_results, ColoredView):
self.update(view_results.tree)
self._colored_view = view_results
else:
self.update(view_results)
finally:
if self.is_root:
if cache_purge_level:
Expand Down Expand Up @@ -1759,6 +1765,10 @@ def _get_view_results(self, view):
result.remove_empty_keys()
elif view == DELTA_VIEW:
result = self._to_delta_dict(report_repetition_required=False)
elif view == COLORED_VIEW:
result = ColoredView(t2=self.t2, tree_result=self.tree, compact=False)
elif view == COLORED_COMPACT_VIEW:
result = ColoredView(t2=self.t2, tree_result=self.tree, compact=True)
else:
raise ValueError(INVALID_VIEW_MSG.format(view))
return result
Expand Down Expand Up @@ -1899,6 +1909,11 @@ def affected_root_keys(self):
result.add(root_key)
return result

def __str__(self):
if hasattr(self, '_colored_view') and self.view in {COLORED_VIEW, COLORED_COMPACT_VIEW}:
return str(self._colored_view)
return super().__str__()


if __name__ == "__main__": # pragma: no cover
import doctest
Expand Down
Loading