Skip to content
All work

01Case study

Property-portal publishing pipeline

Publishing a Cairo real-estate group’s CRM inventory to a major property portal, over an integration where the portal never pushes anything back.

The problem
Agents typed every listing twice — once in the CRM, again on a property portal that never reports anything back.
What I built
A publishing pipeline: a readiness check that lists every blocker at once, queued publish jobs, rate-limit handling, and an hourly sweep that reads the portal’s state back.
The result
Listings are written once, in the CRM. A refused publish names the reason and the field to fix.

Sole author: 20,665 lines across 96 files, 35 of them test files — plus 10,923 of 10,928 lines of the listing screens’ front end.

Problem

The group's CRM already held the inventory: every unit, its photographs, its permit paperwork, its price. A major property portal held the same listings a second time, because agents typed them in again by hand. Two copies drift the moment either one is touched, and afterwards nobody can say which one is true.

I built the pipeline that publishes CRM inventory to the portal, so a listing is written once, in the CRM.

Constraint

The portal sends no webhooks. I can publish, unpublish, delete, and ask for the current state of a listing, but nothing ever arrives unprompted. When a listing expires, when a permit is rejected, when a moderator takes something down, the portal knows and I do not. Local state is a guess about a remote system I cannot observe, and a guess that is never corrected turns into a lie.

The portal also rate limits, answering 429 with a retry_after, and publishing a whole project's worth of units will reach that limit. The agents are not developers, so a refused publish must tell them what to fix.

Approach

With no push channel available, I added a pull one. An hourly reconcile sweep, written as a scheduled command, reads the portal's state for the listings we track and corrects the local PublicationState to match. The publication enum has seven states: draft, pending permit, active, expired, rejected, unpublished and failed. The sweep is what makes expired and rejected reachable, because only the portal knows about those transitions. Five queued jobs carry the work: publish, unpublish, delete, sync and reconcile. The client is rate-limit-aware and honours retry_after on a 429. The sync path takes a lockForUpdate, because two concurrent publishes of the same listing would otherwise create a duplicate property on the portal and burn the reference.

Before any of that runs, a readiness service decides whether the listing can be published, and reports every blocker rather than failing on the first one. A 22-case reason enum covers the conditions: missing agent, unmapped property type, unavailable location, size out of range, missing or duplicated permit number, too many images, unavailable amenity, and so on. Each reason also names the form field the agent has to fix, so the interface can jump straight to the right tab.

That enum used to be larger. Fourteen further reason cases were deleted once it was clear the form-request validation already enforced exactly the same rules on write: purpose, reference, title, description, price, rent frequency, room ranges. A listing could not reach the publish step with any of them wrong, so those codes were unreachable. Rejecting a listing for one of them now surfaces as a validation error on write, not as a readiness problem found later. Deleting unreachable cases kept the enum honest.

Trade-off

The hourly sweep means local state can be up to an hour stale: a listing the portal expired at ten past the hour still reads as active until the next run. I accepted it because the portal offers no push mechanism, and a tighter schedule buys freshness with rate-limit budget better spent on publishing.

The readiness check also costs an extra pass over the listing before every publish. That is the price of listing everything wrong in one go instead of only the first thing.

Outcome

Agents no longer retype listings into the portal, the CRM is the only place a listing is written, and a refused publish names both the reason and the field to fix. A state we get wrong is wrong for an hour, not indefinitely.

100%
AuthorshipSole author of the subsystem
Lines
20,665
Test files
35
Readiness reasons
22
Publication states
7
Queued jobs
5
Property portal publishing pipelineA listing edit enters ReadinessService, a gate with 22 possible blocking reasons: on failure it returns the blockers and nothing is sent. A passing listing goes to PayloadBuilder, then to a queued PublishJob, then through a rate-limit-aware client to the external portal API. A 429 with retry_after sends the job back to the queue with backoff. lockForUpdate on the sync path stops two concurrent publishes from creating a duplicate property. Because the portal pushes nothing back, an hourly scheduled reconcile sweep reads portal state and corrects the local PublicationState.Listing editedproperty updatedReadinessServicegate · 22 blocking reasonsblockednothing is sentblockers returned tothe editorpasses gatePayloadBuilderportal-shaped payloadPublishJobqueuedlockForUpdateone publish at a timeRateLimitAwareClientExternal Portal APIpushes nothing back429 · retry_after · backoffreads portalHourly reconcile sweepscheduled commandPublicationStatecorrected locallyThe portal pushes nothing back —the sweep is the only feedback.happy pathfailure · retry

Screens

  • Listing table with a publish-status column showing Active, Rejected, Failed and Not set badges, and an error column with the portal’s rejection reasons, such as an invalid permit number.
    Every listing’s publish state, with the portal’s own rejection reason beside it — what an agent reads instead of a stack trace. Portal name blurred.
  • Publish-state panel for a single listing: status Active, the portal’s property ID (blurred) and the published-at time, under the note "Mirrored from the last sync — sends no webhooks".
    The state on one listing is a mirror, and says so: the portal sends no webhooks, so this is what the last reconcile sweep found.
ReadinessService.phpphp
/**
 * Collect EVERY blocker, not just the first.
 *
 * Failing fast would make an agent fix one field, retry, and fail on the
 * next. The publish is attempted only against a clean result.
 */
public function check(Listing $listing): ReadinessResult
{
    $problems = [];

    foreach ($this->rules as $rule) {
        if (($reason = $rule->evaluate($listing)) !== null) {
            // The reason knows its form field, so the UI can deep-link
            // the agent to the right tab.
            $problems[] = new ReadinessProblem($reason);
        }
    }

    return new ReadinessResult($problems);
}

Paraphrased from a private repository — names changed, logic preserved.

One publish, start to finish

The agent got their answer in 41 milliseconds. Everything that matters happened afterwards.

POST /units/8412/publish202 Accepted · 41msthe caller is already gone
The request ends here
  1. +0ms

    PublishListingJob queued

    Dispatched to the queue rather than run inline, so the agent's request is not holding a connection open across a third-party API call.

  2. +180ms

    Readiness check — 22 reasons evaluated

    Every blocker is reported at once, not just the first, and each one names the form field to fix. Fourteen further cases were deleted once validation made them unreachable.

  3. +240ms

    POST /listings → 429 Too Many Requests

    Publishing a whole project’s worth of units reaches the portal’s rate limit. The response carries retry_after: 30.

  4. +30s

    Backoff — the client waits what it was told to wait

    Not a fixed sleep and not an immediate retry. The portal named a number; honouring it is the difference between a queue that drains and one that gets throttled harder.

  5. +30.4s

    Retry → 201 Created

    The remote reference is written back under lockForUpdate, because two concurrent publishes of the same listing would otherwise create a duplicate on the portal and burn the reference.

  6. +1h

    ReconcileListingsCommand runs

    The portal sends no webhooks. It cannot tell us anything, so once an hour we ask it instead — the only way expired and rejected are reachable at all.

  7. +1h

    Portal says expired — local state corrected

    PublicationState moves active → expired. A guess that is never corrected turns into a lie, and this is the sweep that stops it becoming one.

An hour of staleness, accepted on purpose. The portal offers no push channel, and a tighter schedule buys freshness with rate-limit budget that is better spent publishing.

Need something like this built?

Taking freelance work · ~20h/week.

Start a project