We are gonna’ look into cloning Django model instances, applicability of Iterator and Visitor patterns, and a little bit into Django models metadata. If you want to look into final code - visit github repository - https://github.com/regqueryvalueex/django-relations-iterator
So, what’s the issue?
Lately, working with some Django code I faced a need to implement proper cloning feature. A brief search on the internet gave me some packages for cloning, but I wasn’t satisfied with those. So it’s time to write some code by myself.
Simplest clone in Django looks like this
# models.py
class Meeting(models.Model):
title = models.CharField(max_length=200)
# clone
instance = Meeting.objects.last()
instance.pk = None
instance.save()
This will create new row in database with the same data as original instance have. However if you want also clone some related objects, you may have to manually go over all related objects and clone them as well. Also don’t forget to set correct foreign key value, so you will have correct hierarchy
# models.py
class Meeting(models.Model):
title = models.CharField(max_length=200)
class Participation(models.Model):
meeting = models.ForeignKey('Meeting', on_delete=models.CASCADE, related_name='participations')
# clone
def clone(instance):
instance.pk = None
instance.save()
return instance
instance = Meeting.objects.last()
related_objects = instance.participations.all()
cloned_instance = clone(instance)
for related_object in related_objects:
related_object.meeting = cloned_instance
clone(related_object)
Well, it works, kinda. Lets summarize issues with this
- requires to manually hardcode all cloning process for all relations
- will copy all the data from original instances, event if you don’t need this
- we don’t have any convenient or clear way to controll process for different instances
- adding new models to hierarchy will require us to add new code to this
But what exactly do we want from this feature?
- must provide automatic way to walk over hierarchy
- must provide possibility to control clone process for different instances
- must provide control over relation that will be cloned
- must be reusable for different models and hierarchies
Lets see what we can do…
Step 1. Setup test models
from django.conf import settings
from django.db import models
class Meeting(models.Model):
title = models.CharField(max_length=200)
time = models.DateTimeField(null=True, blank=True)
participants = models.ManyToManyField(
settings.AUTH_USER_MODEL,
through='Participation',
blank=True
)
class Participation(models.Model):
meeting = models.ForeignKey(
'Meeting',
on_delete=models.CASCADE,
related_name='participations'
)
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.CASCADE,
related_name='participations'
)
class Invitation(models.Model):
status = models.CharField(max_length=20, default='sent)
participation = models.ForeignKey(
'Participation',
on_delete=models.CASCADE,
related_name='invitations'
)
class Comment(models.Model):
meeting = models.ForeignKey(
'Meeting',
on_delete=models.CASCADE,
related_name='comments'
)
user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
description = models.TextField(max_length=3000)
Step 2. Determine and configure model relations
Django models have an interface for model metadata, we can read info about relations there
print(Meeting._meta.related_objects)
# (<ManyToOneRel: meetings.participation>, <ManyToOneRel: meetings.comment>)
print(Participation._meta.related_objects)
# (<ManyToOneRel: meetings.invitation>,)
print([relation.get_accessor_name() for relation in Meeting._meta.related_objects])
# ['participations', 'comments']
Method get_accessor_name simply returns a name, that can be used to access related objects, also we can retrieve corresponding relation field with that field
print(Meeting._meta.get_field('participations'))
<ManyToOneRel: meetings.participation>
print(Meeting._meta.get_field('comments'))
<ManyToOneRel: meetings.comment>
Good, so considering this we can create simple configuration, using relation names. Lets say, we want to include Meeting, Participation, and Invitation, but Comment must be excluded.
# Simple config
structure = {
'participations': {
'invitations': {} # No relations - empty dict
} # `comments` isn't here
}
Step 3. Collecting instances accordingly to the config
# Let's create instances first
tom = User.objects.create(username='Tom')
jerry = User.objects.create(username='Jerry')
meeting = Meeting.objects.create(title='dinner')
tom_participation = Participation.objects.create(user_id=tom.id, meeting_id=meeting.id)
jerry_participation = Participation.objects.create(user_id=jerry.id, meeting_id=meeting.id)
Invitation.objects.create(user_id=jerry.id, meeting_id=meeting.id)
Comment.objects.create(user_id=jerry.id, meeting_id=meeting.id)
Now, when we have some data, we can work with this. So, considering the nature of the data, we can use recursion to iterate over our config and put data, obtained from db, into dict. We can tell, that result must look like a tree, so lets create a couple of classes and make some basic tree
import typing
from django.db.models import Model, ManyToManyRel, ManyToOneRel, OneToOneRel
Relation = typing.Union[ManyToManyRel, ManyToOneRel, OneToOneRel]
RelationTree = typing.Dict['TreeNode', typing.Dict[Relation, 'RelationTree']]
RelationTreeConfig = typing.Dict[str, 'RelationTreeConfig']
class TreeNode:
def __init__(
self,
*,
instance: Model,
parent: typing.Optional['TreeNode'] = None,
relation: typing.Optional[Relation] = None
):
self.instance = instance
self.parent = parent
self.relation = relation
@property
def model_class(self):
return type(self.instance)
def __hash__(self):
return hash(f'{str(self.model_class)}-{self.instance.id}')
def __repr__(self):
return f'<{type(self).__name__} for {repr(self.instance).strip("<>")}>'
class ConfigurableRelationTree:
def __init__(self, *, root: Model, structure: RelationTreeConfig):
self.root: Model = root
self.structure: RelationTreeConfig = structure
self.tree: RelationTree = self.collect()
def collect(
self,
*,
root_node: typing.Optional[TreeNode] = None,
structure: typing.Optional[RelationTreeConfig] = None
) -> RelationTree:
if not root_node:
root_node = self.get_node(instance=self.root)
root = root_node.instance
structure = structure if structure is not None else self.structure
subtree = {}
tree = {root_node: subtree}
for sub_relation_name, substructure in structure.items():
sub_relation = root._meta.get_field(sub_relation_name)
related_instances = self._get_related_instances(instance=root, relation=sub_relation)
subtree[sub_relation] = {}
for related_instance in related_instances:
node = self.get_node(instance=related_instance, relation=sub_relation, parent=root_node)
subtree[sub_relation].update(
self.collect(root_node=node, structure=substructure)
)
return tree
def _get_related_instances(
self,
*,
instance: Model,
relation: Relation
) -> typing.List[Model]:
accessor_name = relation.get_accessor_name()
if relation.one_to_one:
instance = getattr(instance, accessor_name, None)
related_instances = [instance] if instance is not None else []
else:
related_instances = list(getattr(instance, accessor_name).all())
return related_instances
def get_node(
self,
*,
instance: Model,
parent: typing.Optional[TreeNode] = None,
relation: typing.Optional[Relation] = None
) -> TreeNode:
return TreeNode(
instance=instance,
parent=parent,
relation=relation,
)
Lets see what do we have:
from pprint import pprint
tree = ConfigurableModelTree(root=meeting, structure=structure)
pprint(tree._tree)
# {
# <TreeNode for Meeting: Meeting object (1)>: {
# <ManyToOneRel: meetings.participation>: {
# <TreeNode for Participation: Participation object (1)>: {
# <ManyToOneRel: meetings.invitation>: {
# <TreeNode for Invitation: Invitation object (1)>: {}
# }
# },
# <TreeNode for Participation: Participation object (2)>: {
# <ManyToOneRel: meetings.invitation>: {}
# }
# }
# }
# }
Good, seems like we have correct structure. So how we gonna use it? Here, where’s patterns come to our help
Step 4. Get familiar with Iterator and Visitor patterns
Fortunately, programmers around the world have been working with data for many years and have many good solutions to solve almost any question. Our question is -
How to create reusable solution to iterate over our tree structure?
Answer is - Iterator pattern
Iterator is a behavioral design pattern that lets you traverse elements of a collection without exposing its underlying representation (list, stack, tree, etc.).
Our other question:
How to create a reusable solution to operate with tree nodes?
And the answer for that - Visitor pattern
Visitor is a behavioral design pattern that lets you separate algorithms from the objects on which they operate.
And what event better - these patterns work great together.
Step 5. Implement simple Iterator and Visitor
Python have built-in iterator protocol, so we can simply just use that. Simple implementation of __iter__ method, that returns generator will be enough.
import typing
from abc import ABC, abstractmethod
# I will use abstract class to define interface
class AbstractRelationTreeIterator(ABC):
@abstractmethod
def get_iterator(self, tree: typing.Optional[RelationTree] = None):
pass
def __iter__(self):
return self.get_iterator()
class RelationTreeIterator(AbstractRelationTreeIterator):
def __init__(self, tree: ConfigurableRelationTree):
self.tree = tree
def get_iterator(self, tree: typing.Optional[RelationTree] = None):
tree = tree if tree is not None else self.tree.tree
for node, subtree in tree.items():
yield node
for relation, subnodes in subtree.items():
# since we iterate over tree, we need recursion here as well
yield from self.get_iterator(subnodes)
For visitor, it’s even simpler:
class AbstractVisitor(ABC):
@abstractmethod
def visit(self, node: TreeNode):
pass
class CloneVisitor(AbstractVisitor):
def visit(self, node: TreeNode):
node.instance.pk = None
if node.parent is not None:
parent_joining_column, instance_joining_column = node.relation.get_joining_columns()[0]
setattr(
node.instance,
instance_joining_column,
getattr(node.parent.instance, parent_joining_column)
)
node.instance.save()
Method get_joining_columns will return columns, that involved in relation between models. For Meeting and Participation it will return (('id', 'meeting_id'),) , and that exactly what we need, since participation.meeting_id = meeting.id is a correct foreign key assignment.
Step 6. See, how it works
So, how do you use visitor and iterator together. Well you just iterate over your data structure using iterator and use visitor for every item:
tree = ConfigurableRelationTree(root=meeting, structure=structure)
visitor = CloneVisitor()
for node in RelationTreeIterator(tree):
visitor.visit(node)
pprint(tree.tree)
# {
# <TreeNode for Meeting: Meeting object (3)>: {
# <ManyToOneRel: meetings.participation>: {
# <TreeNode for Participation: Participation object (5)>: {
# <ManyToOneRel: meetings.invitation>: {
# <TreeNode for Invitation: Invitation object (3)>: {}
# }
# },
# <TreeNode for Participation: Participation object (6)>: {
# <ManyToOneRel: meetings.invitation>: {}
# }
# }
# }
# }
As you can see, we have different ids here, so new instances was created in database for this hierarchy.
Step 7. Make cloning process configurable
Its very common, that you need to have control over cloned data. For instance, in our case, we probably don’t want to have the same time in cloned instance. And maybe we want to add some extra text to meeting title, like f'{original_title}-COPY'.
Classical Visitor patters suggests us to implement method in our class TreeNode, that will accept a Visitor instance and will call a proper implementation. That is a good way to handle different requirements for different classes, however I want to keep ConfigurableRelationTree and TreeNode untouched, and rely on other tools. Fairly often because different languages have some powerful tools, they allow to simplify some classical patterns. For example First class functions can simplify Strategy pattern or in some cases event replace it.
For our case, we can use functools.singledispatch
from functools import singledispatch
@singledispatch
def customize(instance):
pass
@customize.register
def _(instance: Meeting):
instance.title = f'{instance.title}-COPY'
instance.time = None
# Changed ClonedVisitor
class CloneVisitor(AbstractVisitor):
def visit(self, node: TreeNode):
node.instance.pk = None
if node.parent is not None:
parent_joining_column, instance_joining_column = node.relation.get_joining_columns()[0]
setattr(
node.instance,
instance_joining_column,
getattr(node.parent.instance, parent_joining_column)
)
# added customize call
customize(node.instance)
node.instance.save()
Let’s see now how it works
structure = {
'participations': {
'invitations': {}
}
}
meeting = Meeting.objects.get(title='dinner')
tree = ConfigurableRelationTree(root=meeting, structure=structure)
visitor = CloneVisitor()
for node in RelationTreeIterator(tree):
visitor.visit(node)
print(meeting.title)
# 'dinner-COPY'
print(meeting.time)
# None
Note, that our structure is mutated and we actually overwrite original tree with cloned tree. That’s important, because if you want to use original instances, you need to initialize them again after clone.
Conclusion
Now we have a reusable approach to clone instance hierarchies in Django. However it’s not limited by only clone feature. We actually can use it for other needs and all we need to do most of the time is just create new Visitor implementation, and, maybe, new Iterator implementation. Lets create a quick example, that will print path to our nodes:
class PathPrintVisitor(AbstractVisitor):
def visit(self, node: TreeNode):
print(list(reversed(self.get_path(node))))
def get_path(self, node: TreeNode):
path = [node]
if node.parent:
path.extend(self.get_path(node.parent))
return path
visitor = PathPrintVisitor()
for node in RelationTreeIterator(tree):
visitor.visit(node)
# [<TreeNode for Meeting: Meeting object (5)>]
# [<TreeNode for Meeting: Meeting object (5)>, <TreeNode for Participation: Participation object (9)>]
# [<TreeNode for Meeting: Meeting object (5)>, <TreeNode for Participation: Participation object (9)>, <TreeNode for Invitation: Invitation object (5)>]
# [<TreeNode for Meeting: Meeting object (5)>, <TreeNode for Participation: Participation object (10)>]
You may visit my github repo for this implementation and look into the code if you want to - https://github.com/regqueryvalueex/django-relations-iterator