We define the transaction mainly based on the DUT [Design Under Test – RTL design] interface for the complete testbench infrastructure[Verification Environment], irrespective of the scoreboard. So, the transactions could be AHB/AXI/APB/USB transactions, based on the DUT interface. If the DUT follows any proprietary/custom protocol, then you may want to follow the protocol and define the transaction accordingly.
The scoreboard is one of the transactors[TB components – Generator, Driver, Monitor, Scoreboard, etc] of the verification environment. It basically collects the transactions[Packets] from the DUT reference model and DUT RTL and compares them. So, it compares AHB packets with AHB packets or AXI packets with AXI packets, etc. If the comparison fails, we need to rerun the simulation with waveform and debug the simulation failures.
If the DUT is a networking IP like a router, then the scoreboard can directly compare the router output packet with the input packet directly, without having any reference model, because there is no data transformation in the DUT – router [Output is same as Input]. But the DUTs like processors/controllers which process and transform the data demand behavioural TLM [Transaction Level Model – DUT reference model].
In SystemVerilog, you can use class data type to define any kind of transaction. Also, using OOP features like inheritance and polymorphism, you could create different variations/flavours of the transactions and verify the DUT, by generating different kinds of random scenarios. We always define the transactions upfront based on the DUT interfaces, before creating the testbench infrastructure[Verification Environment]. Sometimes, we may need to plan how best we can configure the testbench. So, every TB component may need some additional configuration transactions too.
One really needs to understand how to pre-plan everything[Transactions, TB components, TB configuration, Reference Model, Coverage model, etc] properly while creating the testbench architecture as part of the verification Planning phase. I have explained this verification planning process and SystemVerilog TB coding in detail as part of our online course VLSI Verification