1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
.\" -*- nroff -*-
.\"
.TH IBV_POST_SEND 3 2006-10-31 libibverbs "Libibverbs Programmer's Manual"
.SH "NAME"
ibv_post_send \- post a list of work requests (WRs) to a send queue
.SH "SYNOPSIS"
.nf
.B #include <infiniband/verbs.h>
.sp
.BI "int ibv_post_send(struct ibv_qp " "*qp" ", struct ibv_send_wr " "*wr" ,
.BI " struct ibv_send_wr " "**bad_wr" );
.fi
.SH "DESCRIPTION"
.B ibv_post_send()
posts the linked list of work requests (WRs) starting with
.I wr
to the send queue of the queue pair
.I qp\fR.
It stops processing WRs from this list at the first failure (that can
be detected immediately while requests are being posted), and returns
this failing WR through
.I bad_wr\fR.
.PP
The argument
.I wr
is an ibv_send_wr struct, as defined in <infiniband/verbs.h>.
.PP
.nf
struct ibv_send_wr {
.in +8
uint64_t wr_id; /* User defined WR ID */
struct ibv_send_wr *next; /* Pointer to next WR in list, NULL if last WR */
struct ibv_sge *sg_list; /* Pointer to the s/g array */
int num_sge; /* Size of the s/g array */
enum ibv_wr_opcode opcode; /* Operation type */
int send_flags; /* Flags of the WR properties */
uint32_t imm_data; /* Immediate data (in network byte order) */
union {
.in +8
struct {
.in +8
uint64_t remote_addr; /* Start address of remote memory buffer */
uint32_t rkey; /* Key of the remote Memory Region */
.in -8
} rdma;
struct {
.in +8
uint64_t remote_addr; /* Start address of remote memory buffer */
uint64_t compare_add; /* Compare operand */
uint64_t swap; /* Swap operand */
uint32_t rkey; /* Key of the remote Memory Region */
.in -8
} atomic;
struct {
.in +8
struct ibv_ah *ah; /* Address handle (AH) for the remote node address */
uint32_t remote_qpn; /* QP number of the destination QP */
uint32_t remote_qkey; /* Q_Key number of the destination QP */
.in -8
} ud;
.in -8
} wr;
.in -8
};
.sp
.nf
struct ibv_sge {
.in +8
uint64_t addr; /* Start address of the local memory buffer */
uint32_t length; /* Length of the buffer */
uint32_t lkey; /* Key of the local Memory Region */
.in -8
};
.fi
.PP
Each QP Transport Service Type supports a specific set of opcodes, as shown in the following table:
.PP
.nf
OPCODE | IBV_QPT_UD | IBV_QPT_UC | IBV_QPT_RC
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-
IBV_WR_SEND | X | X | X
IBV_WR_SEND_WITH_IMM | X | X | X
IBV_WR_RDMA_WRITE | | X | X
IBV_WR_RDMA_WRITE_WITH_IMM | | X | X
IBV_WR_RDMA_READ | | | X
IBV_WR_ATOMIC_CMP_AND_SWP | | | X
IBV_WR_ATOMIC_FETCH_AND_ADD | | | X
.fi
.PP
The attribute send_flags describes the properties of the \s-1WR\s0. It is either 0 or the bitwise \s-1OR\s0 of one or more of the following flags:
.PP
.TP
.B IBV_SEND_FENCE \fR Set the fence indicator. Valid only for QPs with Transport Service Type \fBIBV_QPT_RC
.TP
.B IBV_SEND_SIGNALED \fR Set the completion notification indicator. Relevant only if QP was created with sq_sig_all=0
.TP
.B IBV_SEND_SOLICITED \fR Set the solicited event indicator. Valid only for Send and RDMA Write with immediate
.TP
.B IBV_SEND_INLINE \fR Send data in given gather list as inline data
in a send WQE. Valid only for Send and RDMA Write. The L_Key will not be checked.
.TP
.B IBV_SEND_IP_CSUM \fR Offload the IPv4 and TCP/UDP checksum calculation.
Valid only when \fBdevice_cap_flags\fR in device_attr indicates current QP is
supported by checksum offload.
.SH "RETURN VALUE"
.B ibv_post_send()
returns 0 on success, or the value of errno on failure (which indicates the failure reason).
.SH "NOTES"
The user should not alter or destroy AHs associated with WRs until
request is fully executed and a work completion has been retrieved
from the corresponding completion queue (CQ) to avoid unexpected
behavior.
.PP
The buffers used by a WR can only be safely reused after WR the
request is fully executed and a work completion has been retrieved
from the corresponding completion queue (CQ). However, if the
IBV_SEND_INLINE flag was set, the buffer can be reused immediately
after the call returns.
.SH "SEE ALSO"
.BR ibv_create_qp (3),
.BR ibv_create_ah (3),
.BR ibv_post_recv (3),
.BR ibv_post_srq_recv (3),
.BR ibv_poll_cq (3)
.SH "AUTHORS"
.TP
Dotan Barak <dotanba@gmail.com>
|
