The Ultimate Tutorial for Django REST Framework: Custom Fields (Part 3)

If you've been keeping a close eye on our blog, you probably didn’t miss the last two parts of this tutorial.

Be sure to catch up with the work we’ve completed in other parts of the series:

In this article, I’ll show you how to add a dynamic fields to our serializer model. Dynamic fields are fields that aren’t part of our database and their value is calculated on a regular basis.

I use the amazing pendulum library for datetime service. I also recommend using ipython to make ./manage.py shell even more awesome

Do you remember that example application we’ve been working on in the last two posts in this series? Just like in the previous pieces, today I wanted to continue using the example of an app for managing items we borrow to our friends.

Let's assume that we want to display information next to the name of our friends whether they’re due to give us back something.

Downloading this information in our data model (see my first article) will look like this. Let’s say I want to know if my Friend is keeping my items for more than two months:

friend = Friend.objects.get(id=1)
friend.borrowed_set.filter(returned__isnull=True, when__lte=pendulum.now().subtract(months=2)).exists()

This piece of code would work great for a single person. However, if we wanted to display this data for a group of people, we would have inevitably flooded the database with hundreds (or even thousands) of queries. As Raymond Hettinger would say: There must be a better way!

And fortunately, there is. We can use the mechanism of annotation. The following query will add the 'ann_overdue` (you’ll see why not `has_overdue` in a minute) field to all elements of the queryset:

Friend.objects.annotate(
    ann_overdue=models.Case(
        models.When(
            borrowed__returned__isnull=True,
            borrowed__when__lte=pendulum.now().subtract(months=2),
            then=True
        ),
        default=models.Value(False),
        output_field=models.BooleanField()
    )
)

Let's analyze this in detail. I use the Case function which takes any number of When functions as a parameter - in our case, one is enough. Inside When, I enter a condition - if among the Borrowed items for each of my Friends the `returned` field is NULL (borrowed__returned__isnull=True), and the `when` field contains a date older than "two months ago", then the value for that Friend will be True. The default value is False, and the result is to be mapped to the BooleanField field. Simple, right? ;)

You can find out more about conditional expressions in querysets in Django documentation.

Where to put this code?

Normally, if we follow the concept of “fat models” we would need to put the entire logic inside the model. However, that’s not always possible because the models (or at least a part of them) can come from third-party apps. I would like to address both cases - pick one that suits your app.

"Fat models" case

In this case, we have full control over the models, so we can do this by the book:

models.py

class FriendQuerySet(models.QuerySet):
    def with_overdue(self):
        return self.annotate(
            ann_overdue=models.Case(
                models.When(
                    borrowed__when__lte=pendulum.now().subtract(months=2),
                    then=True),
                default=models.Value(False),
                output_field=models.BooleanField()
            )
        )

class Friend(OwnedModel):
    name = models.CharField(max_length=100)

    objects = FriendQuerySet.as_manager()

    @property
    def has_overdue(self):
        if hasattr(self, 'ann_overdue'): # in case we deal with annotated object
            return self.ann_overdue
        return self.borrowed_set.filter(
            returned__isnull=True, when=pendulum.now().subtract(months=2)
        ).exists()

views.py

class FriendViewset(viewsets.ModelViewSet):
    queryset = models.Friend.objects.with_overdue()
    serializer_class = serializers.FriendSerializer
    # and so on...

serializers.py

class FriendSerializer(serializers.ModelSerializer):
    owner = serializers.HiddenField(
        default=serializers.CurrentUserDefault()
    )

    class Meta:
        model = models.Friend
        fields = ('id', 'name', 'has_overdue')

Third-party models case

In this case we can’t modify models, so the view would have to have its `get_queryset` method overwritten and serializers.py would have the `has_overdue` logic:

views.py

class FriendViewset(viewsets.ModelViewSet):
    queryset = models.Friend.objects.all()
    serializer_class = serializers.FriendSerializer
    # and so on…

    def get_queryset(self):
        return super().get_queryset().annotate(
            ann_overdue=models.Case(
                models.When(
                    borrowed__when__lte=pendulum.now().subtract(months=2),
                    then=True),
                default=models.Value(False),
                output_field=models.BooleanField()
            )
        )

serializers.py

class FriendSerializer(serializers.ModelSerializer):
    owner = serializers.HiddenField(
        default=serializers.CurrentUserDefault()
    )
    has_overdue = serializers.SerializerMethodField()

    class Meta:
        model = models.Friend
        fields = ('id', 'name', 'has_overdue')

    def get_has_overdue(self, obj):
        if hasattr(obj, 'ann_overdue'):
            return obj.ann_overdue
        return obj.borrowed_set.filter(
            returned__isnull=True, when=pendulum.now().subtract(months=2)
        ).exists()

Wrap up

Now as you see, I define a `has_overdue` field (or property), put it in serializer’s Meta.fields attribute and, if need be, explain to DRF how to get the value. Notice how I check for `ann_overdue` attribute. That way I get very universal code - if the annotation was used, the value is already calculated and we can re-use it. If it wasn’t - well, we need to do the heavy lifting ourselves.

BTW. Have a look at our blog to learn more about building Django models.

Impact on DB

Now how this changes the db hit count? I’ve prepared sample dataset that had 1000 Friends, 10000 Belongings distributed randomly past the last six months and did the test (print results cut out for clarity):

In [1]: from django.db import connection

In [2]: from core.models import Friend

In [3]: for f in Friend.objects.all(): ...: print(f.has_overdue) ...:

In [4]: len(connection.queries) Out[4]: 1000 

another try using with_overdue method:

In [1]: from django.db import connection

In [2]: from core.models import Friend

In [3]: for f in Friend.objects.with_overdue(): ...: print(f.has_overdue) ...:

In [4]: len(connection.queries) Out[4]: 1 

Conclusion

So there you have it. For “fat models”, you just need to put the method or property name in the serializer’s Meta.fields attribute. An extra benefit you’ll be getting here is that the manager and `has_overdue` property can also be used in admin panel so you get two serious improvements in one slim package.

If your models come from a third-party app, you should put the required logic inside the serializer as it can be used by more than one view. Unfortunately, in our case the logic needs to operate (annotate) the queryset, so the logic lands inside a view. But if it needs to be reused in other views, you can make a mixin just for the `get_queryset` method.

And remember to always anticipate the impact on the database and consider using `select_related` and/or `prefetch_related` when getting the queryset.

I hope this post helps you work with Django REST Framework efficiently. Stay tuned - in the next article, I’ll be taking a closer look at pagination and filtering.